Skip to content

epic: migration of workspaces between shards #3498

New issue
New issue
Open
Epic
14 of 17 issues completed
Open
epic: migration of workspaces between shards#3498
Epic
14 of 17 issues completed
Assignees
ntnn
Labels
area/shardingIssues or PRs related to sharding changeskind/featureCategorizes issue or PR as related to a new feature.
Milestone
TBD

Description

@embik
embik
opened on Jul 29, 2025

Feature Description

For a growing kcp instance, migrating logical clusters between shards might eventually become necessary, especially when single shard environments evolve to multi-shard ones. As such, it should be possible to migrate workspaces from one shard to another, either online or at the very least offline.

Proposed Solution

Unclear right now. Some mechanism to put logical clusters into maintenance, fetch all data stored in the underlying data store, send it to the other shard.

Alternative Solutions

No response

Want to contribute?

  • I would like to work on this issue.

Additional Context

No response

PRs

Activity

  1. embik
    added
    kind/featureCategorizes issue or PR as related to a new feature.
    on Jul 29, 2025
  2. kcp-ci-bot
    added this to kcpon Jul 29, 2025
  3. github-project-automation
    moved this to New in kcpon Jul 29, 2025
  4. embik
    added
    area/shardingIssues or PRs related to sharding changes
    on Jul 31, 2025
  5. embik
    added this to the TBD milestone on Aug 4, 2025
  6. embik
    added the
    Feature
    issue type on Aug 5, 2025
  7. mirzakopic
    moved this to Backlog in Platform Mesh & Kube Projects - Backlogon Sep 4, 2025
  8. mirzakopic
    added this to Platform Mesh & Kube Projects - Backlogon Sep 4, 2025

  9. mjudeikis commented on Dec 22, 2025

    @mjudeikis
    mjudeikis
    on Dec 22, 2025
    Contributor

    We been chatting about this. There should be some ways to achieve this using etcd client and maybe dedicated management vw.

    1. Shards put to maintenance mode (read only?)
    2. Workspace move iniciated via VW
    3. Update frontproxy index (how, maybe a simple restart of both shards and fp for now)
    4. Shards taken out of maintenance (restart?)

    Basically moving from one shard to another. I propose we start with simpliest approach:

    1. Shards maintenance mode (Read only)
    2. dedicated VW which has access to both shards.
    3. Restart all the things to commit.

    Current snip of shard etcd keys:

    /registry/core/secrets/root/kcp-system/topology.kcp.io
/registry/core/serviceaccounts/2zvtcam3cl4vycy3/default/default
/registry/core/serviceaccounts/2zvtcam3cl4vycy3/kube-system/default
/registry/core/serviceaccounts/30pb2119n7xgh3kq/default/default
/registry/core/serviceaccounts/30pb2119n7xgh3kq/kube-system/default
/registry/core/serviceaccounts/root/default/default
/registry/core/serviceaccounts/root/kcp-system/default
/registry/core/serviceaccounts/system:admin/default/default
/registry/core/serviceaccounts/system:admin/kube-public/default
/registry/core/serviceaccounts/system:admin/kube-system/default
/registry/core/serviceaccounts/system:shard/default/default
/registry/rbac.authorization.k8s.io/clusterrolebindings/2zvtcam3cl4vycy3/workspace-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/30pb2119n7xgh3kq/workspace-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:apiexport:kcp-admin:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:apiexport:tenancy:bind
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:apiexport:tenancy:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:apiexport:topology:bind
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:apiexport:topology:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:tenancy:reader
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:authenticated:universal-workspacetype-use
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:configured-group:home-clusterworkspace-create
/registry/rbac.authorization.k8s.io/clusterrolebindings/root/system:kcp:tenancy:authenticated:access
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/cluster-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:basic-user
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:attachdetach-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:certificate-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:clusterrole-aggregation-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:cronjob-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:daemon-set-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:deployment-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:disruption-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:endpoint-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:endpointslice-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:endpointslicemirroring-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:ephemeral-volume-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:expand-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:generic-garbage-collector
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:horizontal-pod-autoscaler
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:job-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:legacy-service-account-token-cleaner
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:namespace-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:node-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:persistent-volume-binder
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:pod-garbage-collector
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:pv-protection-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:pvc-protection-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:replicaset-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:replication-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:resource-claim-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:resourcequota-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:root-ca-cert-publisher
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:route-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:selinux-warning-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:service-account-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:service-cidrs-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:service-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:statefulset-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:ttl-after-finished-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:ttl-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:validatingadmissionpolicy-status-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:controller:volumeattributesclass-protection-controller
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:discovery
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kcp:admin:cluster-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kcp:external-logical-cluster-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kcp:logical-cluster-admin
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kcp:tenancy:workspace-bootstrapper
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kube-controller-manager
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kube-dns
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:kube-scheduler
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:monitoring
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:node
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:node-proxier
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:public-info-viewer
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:service-account-issuer-discovery
/registry/rbac.authorization.k8s.io/clusterrolebindings/system:admin/system:volume-scheduler
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:apiexport:kcp-admin:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:apiexport:tenancy:bind
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:apiexport:tenancy:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:apiexport:topology:bind
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:apiexport:topology:maximal-permission-policy
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:home-clusterworkspace-create
/registry/rbac.authorization.k8s.io/clusterroles/root/system:kcp:universal-workspacetype-use
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/cluster-admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/edit
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:aggregate-to-admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:aggregate-to-edit
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:aggregate-to-view
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:auth-delegator
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:basic-user
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:certificatesigningrequests:nodeclient
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:kube-apiserver-client-approver
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:kubelet-serving-approver
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:certificates.k8s.io:legacy-unknown-approver
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:attachdetach-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:certificate-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:clusterrole-aggregation-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:cronjob-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:daemon-set-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:deployment-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:disruption-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:endpoint-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:endpointslice-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:endpointslicemirroring-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:ephemeral-volume-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:expand-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:generic-garbage-collector
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:horizontal-pod-autoscaler
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:job-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:legacy-service-account-token-cleaner
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:namespace-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:node-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:persistent-volume-binder
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:pod-garbage-collector
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:pv-protection-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:pvc-protection-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:replicaset-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:replication-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:resource-claim-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:resourcequota-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:root-ca-cert-publisher
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:route-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:selinux-warning-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:service-account-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:service-cidrs-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:service-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:statefulset-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:ttl-after-finished-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:ttl-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:validatingadmissionpolicy-status-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:controller:volumeattributesclass-protection-controller
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:discovery
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:heapster
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kcp:external-logical-cluster-admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kcp:logical-cluster-admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kcp:tenancy:reader
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kcp:tenancy:workspace-bootstrapper
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kcp:workspace:access
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kube-aggregator
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kube-controller-manager
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kube-dns
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kube-scheduler
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:kubelet-api-admin
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:monitoring
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:node
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:node-bootstrapper
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:node-problem-detector
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:node-proxier
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:persistent-volume-provisioner
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:public-info-viewer
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:service-account-issuer-discovery
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/system:volume-scheduler
/registry/rbac.authorization.k8s.io/clusterroles/system:admin/view
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-public/system:controller:bootstrap-signer
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system::extension-apiserver-authentication-reader
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system::leader-locking-kube-controller-manager
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system::leader-locking-kube-scheduler
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system:controller:bootstrap-signer
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system:controller:cloud-provider
/registry/rbac.authorization.k8s.io/rolebindings/system:admin/kube-system/system:controller:token-cleaner
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-public/system:controller:bootstrap-signer
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/extension-apiserver-authentication-reader
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/system::leader-locking-kube-controller-manager
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/system::leader-locking-kube-scheduler
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/system:controller:bootstrap-signer
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/system:controller:cloud-provider
/registry/rbac.authorization.k8s.io/roles/system:admin/kube-system/system:controller:token-cleaner
/registry/tenancy.kcp.io/workspaces/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/2zvtcam3cl4vycy3/foo
/registry/tenancy.kcp.io/workspaces/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/foo
/registry/tenancy.kcp.io/workspacetypes/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/home
/registry/tenancy.kcp.io/workspacetypes/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/organization
/registry/tenancy.kcp.io/workspacetypes/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/root
/registry/tenancy.kcp.io/workspacetypes/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/team
/registry/tenancy.kcp.io/workspacetypes/0eba02d29e6ede981cb44937983138b1d1ebe029131efaa65a8e91af4d0428d9/root/universal
  10. mirzakopic
    moved this from Backlog to No status in Platform Mesh & Kube Projects - Backlogon Jan 23, 2026
  11. ntnn
    self-assigned this
    on May 4, 2026
  12. ntnn
    moved this to ParkingLot in Platform Mesh & Kube Projects - Backlogon May 11, 2026
  13. ntnn
    moved this from ParkingLot to In Progress in Platform Mesh & Kube Projects - Backlogon May 12, 2026

  14. ntnn commented on May 13, 2026

    @ntnn
    ntnn
    on May 13, 2026
    Member

    kcp-dev/enhancements#13

  15. ntnn
    changed the title [-]feature: migration of workspaces between shards[/-] [+]epic: migration of workspaces between shards[/+] on Jun 5, 2026

  16. 6 remaining items

  17. embik commented on Jun 30, 2026

    @embik
    embik
    on Jun 30, 2026
    MemberAuthor

    I know I'm a little late to the party but it's awesome to see progress on this one.

  18. ntnn
    added a sub-issue on Jul 9, 2026
  19. SimonTheLeg
    added a parent issue on Jul 9, 2026
  20. SimonTheLeg
    added a sub-issue on Jul 10, 2026
  21. github-project-automation
    added this to PlatformMesh - RoadMapon Sep 30, 2026
  22. ntnn
    added sub-issues on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/shardingIssues or PRs related to sharding changeskind/featureCategorizes issue or PR as related to a new feature.

Type

Epic

Projects

  • kcp
    Status
    New

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions