feat(podman): relay callbacks through the rootless network namespace

[elezar]

Problem Statement

Rootless Podman sandboxes using slirp4netns cannot connect directly to a
gateway callback listener in the host network namespace. Podman launches
slirp4netns with --disable-host-loopback; QEMU guest validation showed that
listeners bound to loopback, the host default-route address, and 0.0.0.0 were
all unreachable from an OpenShell-style user-defined Podman network.

PR #2492 therefore restricts direct local rootless callbacks to an explicitly
reported pasta helper. Explicit slirp4netns, missing helper metadata, and
unknown helpers fail closed unless the operator configures a genuinely remote
grpc_endpoint.

OpenShell should support local callbacks with rootless slirp4netns without
requiring operators to deploy remote gateway ingress.

Proposed Design

Add a driver-owned, byte-transparent callback relay spanning Podman's rootless
network namespace and the gateway host namespace:

sandbox supervisor
  -> host.openshell.internal:<callback-port>
  -> TCP relay inside `podman unshare --rootless-netns`
  -> mode-0600 Unix socket under $XDG_RUNTIME_DIR/openshell
  -> host-side relay
  -> gateway callback-only listener on IPv4 loopback

The initial proof in PR #2473 used two socat processes:

1. A host-namespace Unix listener forwarding to the gateway loopback listener.
2. A TCP listener launched with podman unshare --rootless-netns, forwarding
   to that Unix socket.

Production support should implement and supervise the relay in OpenShell code
rather than require socat:

• Add a Podman-driver relay manager, or a narrowly scoped relay subcommand
  launched by the driver under podman unshare --rootless-netns.
• Use one relay per driver/network and callback port, not one per sandbox.
• Start and verify the relay before persisted sandboxes resume or new sandboxes
  are created.
• Retry transient forwarding failures, surface terminal setup failures as
  driver preconditions, and clean up child processes and socket files on
  shutdown.
• Report/request only an IPv4 loopback callback listener from the gateway. Do
  not broaden the primary listener.
• Preserve supervisor-to-gateway mTLS end to end; the relay must not terminate
  TLS or inspect application payloads.
• Create the Unix socket in a user-owned runtime directory with mode 0600,
  reject unsafe/stale socket paths, and avoid logging credentials or payloads.
• Keep callback listeners restricted to sandbox-callable gRPC methods.
• Probe podman unshare --rootless-netns support and fail closed when the relay
  topology cannot be established.

Explicitly reported slirp4netns should select this relay. Missing helper
metadata may use it only when the relay capability and reachability can be
verified; absence of rootlessNetworkCmd must not be treated as proof of a
legacy direct-listener topology.

Alternatives Considered

• Bind the gateway to the host default-route address: works with pasta, but
  failed with slirp4netns because host loopback access is disabled.
• Bind the gateway to 0.0.0.0: the diagnostic wildcard listener was also
  unreachable and would unnecessarily broaden the gateway attack surface.
• Set host_gateway_ip: changing the alias target cannot bypass
  slirp4netns host-loopback isolation.
• Require pasta permanently: safe as the current interim behavior, but
  excludes distributions and installations that use slirp4netns.
• Require a remote grpc_endpoint: already supported and appropriate for
  deployments with secure routable ingress, but adds unnecessary
  infrastructure for a single-host local gateway.

Agent Investigation

• Related contract: refactor(compute): define a uniform gateway callback endpoint contract #2215.
• Interim implementation: PR feat(compute): negotiate gateway callback listeners #2492.
• Disposable QEMU tooling: PRs test(e2e): add reusable QEMU infrastructure for E2E tests #2471 and test(e2e): add VM-backed E2E suite runner #2473.
• Fedora 44 / Podman 5.8.1 / pasta reached the host default-route listener.
• CentOS Stream 10 / Podman 6.0.2 / pasta reached the host default-route
  listener.
• Fedora 44 / Podman 5.8.1 / forced slirp4netns reported slirp4netns,
  launched it with --disable-host-loopback, and could reach neither the
  address-specific nor wildcard host listener.
• Ubuntu 24.04 / Podman 4.9.3 omitted rootlessNetworkCmd; the live helper was
  slirp4netns with --disable-host-loopback, and neither host listener was
  reachable.
• The PR test(e2e): add VM-backed E2E suite runner #2473 guest relay made the callback path reachable, demonstrating the
  namespace-spanning relay topology, but intentionally masked the direct
  listener behavior measured above.
• Duplicate searches for "rootless Podman relay", "slirp4netns callback", and
  "rootless network namespace" found no existing issue with this scope.

Acceptance Criteria

• Rootless Podman with explicitly reported slirp4netns establishes local
  supervisor callbacks through the relay.
• Relay startup is readiness-gated and failures prevent sandbox startup
  with an actionable error.
• Supervisor-to-gateway mTLS and callback-only RPC authorization remain
  end to end.
• Relay sockets and processes are securely created, supervised, restarted,
  and cleaned up.
• Rootless pasta retains the direct default-route listener path.
• Rootful Podman retains the exact bridge-gateway listener path.
• Explicitly remote callback endpoints remain unchanged and request no
  local listener.
• Missing helper metadata fails closed unless relay capability and
  reachability are positively verified.
• QEMU E2E coverage includes Fedora and Ubuntu slirp4netns guests plus a
  pasta regression case.
• Architecture, Podman networking, gateway configuration, and
  troubleshooting documentation describe the supported topologies.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[elezar]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: High
Confidence: Medium — the relay topology is validated on Ubuntu 24.04 with
Podman 4.9.3, but listener-purpose deduplication and driver lifecycle semantics
need maintainer confirmation.

Summary

Add a driver-owned, byte-transparent callback relay for rootless Podman
slirp4netns, stacked on PR #2492. The initial implementation will prepare the
shared relay idempotently from GetGatewayListenerRequirements, return a
dedicated callback listener requirement, and fail closed if the relay cannot be
made ready. It will not split CreateSandbox into Create/Start because relay
readiness is driver-wide infrastructure rather than per-sandbox state.

The validated transport is:

sandbox supervisor
  -> host.containers.internal mapped to the inspected network gateway
     (for example, 10.89.0.1)
  -> 0.0.0.0:<callback-port> relay inside
     `podman unshare --rootless-netns`
  -> mode-0600 Unix socket under XDG_RUNTIME_DIR
  -> host-side byte relay
  -> gateway callback-only listener

The wildcard bind exists only inside Podman's rootless network namespace. The
gateway listener remains loopback or otherwise narrowly scoped and
callback-only.

Validated Findings

• Ubuntu 24.04.4 with rootless Podman 4.9.3 uses
  slirp4netns --disable-host-loopback while rootlessNetworkCmd is
  unreported.
• Podman's host-gateway maps the sandbox alias to 10.0.2.15 in this
  topology. A direct callback to the host fails, and a relay in the rootless
  network namespace does not make that blocked slirp host path usable.
• Mapping the alias to the inspected user-defined Podman network gateway
  (10.89.0.1 in the test) and relaying from the rootless network namespace
  succeeds.
• The relay must be ready before sandbox creation. Binding exactly to
  10.89.0.1 fails with EADDRNOTAVAIL because Podman does not materialize
  that bridge address until a container joins the network.
• Binding 0.0.0.0:<callback-port> inside the rootless network namespace
  before sandbox creation succeeds; after the first container materializes the
  bridge, traffic to 10.89.0.1:<callback-port> reaches that listener.
• IP_FREEBIND may permit an exact pre-bind to the future gateway address, but
  it is Linux-specific, requires relay support and additional validation, and
  is optional hardening rather than an initial implementation dependency.
• The proof-of-concept preserved the intended host boundary: a mode-0600 Unix
  socket connected the namespace relay to a host-loopback tagged responder.
  The direct path failed and the relayed path returned the expected tag.

Scope

• proto/compute_driver.proto: clarify that listener-requirement discovery may
  idempotently materialize driver-owned routing prerequisites. Add no new
  sandbox lifecycle RPC in the initial implementation.
• crates/openshell-driver-podman/src/relay.rs (new): own the secure runtime
  directory, Unix listener, host forwarding tasks, child readiness handshake,
  supervision, bounded connection handling, and cleanup.
• crates/openshell-driver-podman/src/main.rs and CLI/config modules: add a
  private relay-child mode invoked with structured arguments under
  podman unshare --rootless-netns; do not invoke a shell or require socat.
• crates/openshell-driver-podman/src/driver.rs and grpc.rs: select the relay
  for explicit slirp4netns; allow missing helper metadata only after positive
  relay capability/readiness checks; map the sandbox callback alias to the
  inspected Podman network gateway for the relay topology; keep pasta,
  rootful, Podman Machine, and remote endpoint paths unchanged; make
  requirement discovery async and idempotent as needed.
• crates/openshell-server/src/lib.rs and listener tests from PR feat(compute): negotiate gateway callback listeners #2492: ensure
  the relay target is a callback-only listener and cannot be silently
  de-duplicated onto a full-primary listener with broader routing authority.
• nix/test-guest/ QEMU coverage: add or extend Fedora and Ubuntu rootless
  slirp4netns callback tests, plus pasta and rootful regression coverage. Reuse
  PRs test(e2e): add reusable QEMU infrastructure for E2E tests #2471/test(e2e): add VM-backed E2E suite runner #2473 after resolving their stack dependency.
• crates/openshell-driver-podman/NETWORKING.md,
  architecture/gateway.md, docs/reference/sandbox-compute-drivers.mdx, and
  .agents/skills/debug-openshell-cluster/SKILL.md: document topology,
  readiness, failure behavior, and troubleshooting.

Implementation Steps

1. Resolve the callback-only relay target. Extend feat(compute): negotiate gateway callback listeners #2492's listener
   materialization only as needed so the relay targets a listener whose purpose
   remains ComputeDriverCallback. A primary loopback or wildcard listener
   must not satisfy the requirement if that would expose the full multiplexed
   API through the relay. Prefer a dedicated loopback address/port when it can
   coexist safely; otherwise introduce the smallest typed callback endpoint
   needed by the requirement contract.
2. Implement a private relay child. Add a hidden Podman-driver subcommand
   launched as podman unshare --rootless-netns <driver> <relay-child> ....
   It binds 0.0.0.0:<callback-port> inside that namespace before any sandbox
   materializes the Podman bridge address, reports readiness over an explicit
   pipe/socket handshake, and forwards opaque bidirectional streams to the
   private Unix socket with half-close and cancellation propagation. Do not
   bind the gateway process itself to a wildcard address.
3. Implement the parent relay manager. Create a user-owned directory under
   XDG_RUNTIME_DIR, reject symlinks/unexpected owners/unsafe permissions,
   create a mode-0600 Unix socket, forward accepted streams to the
   callback-only gateway listener, bound connections and buffers, and remove
   only artifacts whose identity this process established.
4. Prepare the relay during requirement discovery. Make
   GetGatewayListenerRequirements idempotently ensure one relay per
   driver/rootless-network-namespace/port, wait for the namespace listener,
   Unix socket, and parent relay tasks to become locally ready, and then return
   the dedicated callback listener requirement. The gateway materializes that
   host listener before sandbox resume or creation. Concurrent or repeated
   calls must reuse the same healthy relay and roll back partial startup.
5. Use the Podman network gateway as the sandbox destination. Inspect the
   OpenShell user-defined Podman network, map host.containers.internal (or
   the callback alias selected by feat(compute): negotiate gateway callback listeners #2492) to its gateway address, and do not use
   Podman's host-gateway value for this relay topology.
6. Supervise and fail closed. Keep the child and relay tasks alive for the
   driver lifetime. Treat unexpected death as a terminal driver health failure
   (or reject subsequent validate/create calls) rather than falling back to a
   host listener. Ensure gateway bind rejection/startup failure drops the
   managed driver and reaps relay children and sockets.
7. Preserve existing topologies. Keep rootless pasta on the direct
   default-route listener, rootful Podman on the exact bridge gateway, Podman
   Machine on gvproxy loopback, and explicitly remote grpc_endpoint values
   without local requirements. Unknown named helpers remain unsupported.
8. Add tests and failure injection. Cover topology and alias selection,
   idempotency, concurrency, readiness timeout, child exit, filesystem attacks,
   raw-byte fidelity, half-closes, backpressure, cleanup, port collisions,
   listener-purpose isolation, and unchanged topology paths.
9. Add QEMU validation. Make Ubuntu 24.04 with Podman 4.9.3 and missing
   helper metadata a regression case: prove the direct callback fails and the
   relayed mTLS callback through the inspected network gateway succeeds. Also
   run Fedora with explicit slirp4netns, restart/resume behavior, and
   pasta/rootful regressions.
10. Update architecture, operator docs, and diagnostics. Describe the
    relay's trust boundary, prerequisites, observable failures, and
    SELinux/AppArmor troubleshooting without adding a new operator-facing
    configuration field unless implementation proves one necessary.

Test Plan

• Unit tests: topology and callback-alias selection; repeated/concurrent
  prepare; secure runtime-directory and socket creation;
  stale/symlink/ownership rejection; child handshake/timeout/exit; cleanup;
  relay byte fidelity, half-close, cancellation, bounded concurrency, and port
  collisions.
• Integration tests: gateway startup ordering and callback-listener purpose;
  prove sandbox resume/create does not proceed through an unready relay; prove
  primary-only methods remain rejected through the relay target.
• E2E tests: Ubuntu 24.04/Podman 4.9.3 proves the direct path fails and the
  network-gateway relay establishes supervisor mTLS callbacks; Fedora explicit
  slirp4netns and restart/resume succeed; pasta and rootful callbacks remain
  direct.
• Validation: mise run pre-commit, mise run test, relevant Podman/QEMU
  E2E lanes, and mise run ci before PR creation.

Risks & Open Questions

• Getter side effects: Document
  GetGatewayListenerRequirements as an idempotent prepare/ensure operation.
  If maintainers require strict read-only semantics, add a driver-wide
  activation RPC after listener binding; do not split CreateSandbox.
• Listener-purpose isolation: How should a dedicated callback-only target
  coexist with a primary 127.0.0.1:<port> or 0.0.0.0:<port> listener?
  Existing address-coverage de-duplication must not widen the relay's API
  surface.
• Namespace wildcard exposure: The wildcard listener is limited to the
  same-user Podman rootless network namespace, but other containers sharing
  that namespace may reach the callback port. Retain mTLS, expose only
  callback-authorized methods, detect port collisions, and never relay the
  gateway's primary API surface.
• Post-readiness failure: Prefer failing the Podman driver process so the
  gateway observes driver loss. Confirm whether gateway-wide unhealthiness or
  driver-scoped unavailability is the desired blast radius.
• Missing helper metadata: Podman 4 may use the relay only after
  podman unshare --rootless-netns and the complete relay readiness handshake
  succeed. Missing metadata is never evidence of a direct topology.
• Multiple networks and instances: One wildcard listener can cover
  multiple Podman networks that share a rootless network namespace and port.
  Relay identity and ownership must therefore be keyed by the actual namespace
  and port, while socket names must also avoid cross-driver collisions and path
  traversal.
• LSM compatibility: Do not identify children through
  /proc/<pid>/exe; SELinux can return ENOENT across labels. Use an explicit
  readiness handshake and test Fedora enforcing SELinux and Ubuntu AppArmor
  behavior for podman unshare, Unix sockets, and child execution.
• QEMU stack dependency: PR feat(compute): negotiate gateway callback listeners #2492 does not currently contain PR test(e2e): add reusable QEMU infrastructure for E2E tests #2471's
  test-guest tooling. Decide whether test(e2e): add reusable QEMU infrastructure for E2E tests #2471 lands first, the stacked PR carries
  the minimal harness commits temporarily, or E2E lands as a follow-up commit
  after rebasing.
• Optional exact binding: IP_FREEBIND could narrow the namespace listener
  to the future bridge gateway address, but it requires a Linux-specific relay
  implementation and independent kernel/namespace validation. It is deferred
  hardening; wildcard binding inside the isolated rootless namespace is the
  validated initial mechanism.

Documentation Impact

• Update Podman networking and gateway architecture docs with the stable
  topology, startup ordering, authorization boundary, and failure behavior.
• Update published compute-driver documentation with slirp4netns support and
  runtime prerequisites.
• Update debug-openshell-cluster diagnostics for relay readiness, child
  failure, runtime-directory permissions, port collisions, and
  SELinux/AppArmor checks.
• No gateway TOML change is planned. If a new driver config field becomes
  necessary, update docs/reference/gateway-config.mdx in the same branch.

---

Revision 2 — incorporates the Ubuntu 24.04/Podman 4.9.3 proof of concept:
network-gateway aliasing, namespace-only wildcard pre-bind, concrete readiness
ordering, and IP_FREEBIND as optional future hardening.

Revision 1 — initial plan; selects idempotent requirement-time relay
preparation and records callback-listener isolation as the main review
decision.