Skip to content

keyservice client with unix domain sockets broken in 3.10 #1846

Closed

Description

@jnaab

I want run run a sops key service and provide a unix socket via bind mount to containers. This appears broken in 3.10.2 even when running without any bind mounts at all, but worked with 3.9.4.

Setup

sops keyservice --verbose --network unix --address $XDG_RUNTIME_DIR/sops/keyservice

NB: version of the server does not seem to make a difference

sops 3.10.2, broken

.../sops-v3.10.2/sops --verbose --enable-local-keyservice=false --keyservice 'unix:///run/user/1000/sops/keyservice' -d <omitteomittedd>.sops
[CMD]    INFO[0000] Connecting to key service                     address="unix:///run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] original dial target is: "/run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"dns", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"//run/user/1000/sops/keyservice", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel authority set to "%2Frun%2Fuser%2F1000%2Fsops%2Fkeyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel exiting idle mode
2025/04/23 00:00:00 INFO: [core] [Channel #1]Resolver state updated: {
  "Addresses": null,
  "Endpoints": [],
  "ServiceConfig": null,
  "Attributes": null
} ()
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel switches to new LB policy "pick_first"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to TRANSIENT_FAILURE
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
<snip>

There is no activity visible via strace on the server. No connection is attempted.

sops 3.9.4, working

.../sops-v3.9.4/sops --verbose --enable-local-keyservice=false --keyservice 'unix:///run/user/1000/sops/keyservice' -d <omitted>.sops
[CMD]    INFO[0000] Connecting to key service                     address="unix:///run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] original dial target is: "/run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"//run/user/1000/sops/keyservice", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel authority set to "%2Frun%2Fuser%2F1000%2Fsops%2Fkeyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Resolver state updated: {
  "Addresses": [
    {
      "Addr": "/run/user/1000/sops/keyservice",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "/run/user/1000/sops/keyservice",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel switches to new LB policy "pick_first"
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to CONNECTING
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel exiting idle mode
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to CONNECTING
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel picks a new address "/run/user/1000/sops/keyservice" to connect
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to READY
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to READY
[SOPS]   INFO[0000] Data key recovered successfully
[SOPS]   DEBU[0000] Decrypting tree
<snip>

Investigation

Possible change: 060f7a4 (Stop using deprecated grpc.Dial(), use grpc.NewClient() instead., 2025-03-02)

Activity

  1. chirayuk commented on Jul 22, 2025

    @chirayuk

    I'm experiencing this issue as well on v3.10.2 (and v3.10.1). Unix socket fails (tcp works fine). No logs on the server process.

  2. felixfontein commented on Jul 23, 2025

    @felixfontein
    Contributor

    I can confirm that this is caused by 060f7a4.

    When playing around a bit, it seems that

    diff --git a/cmd/sops/main.go b/cmd/sops/main.go
    index f40b181ee..cef927f35 100644
    --- a/cmd/sops/main.go
    +++ b/cmd/sops/main.go
    @@ -2178,22 +2178,26 @@ func keyservices(c *cli.Context) (svcs []keyservice.KeyServiceClient) {
                            continue
                    }
                    addr := url.Host
    -               if url.Scheme == "unix" {
    -                       addr = url.Path
    -               }
    +               addrToUse := addr
                    opts := []grpc.DialOption{
                            grpc.WithTransportCredentials(insecure.NewCredentials()),
    -                       grpc.WithContextDialer(
    -                               func(ctx context.Context, addr string) (net.Conn, error) {
    -                                       return (&net.Dialer{}).DialContext(ctx, url.Scheme, addr)
    -                               },
    -                       ),
    +               }
    +               if url.Scheme == "unix" {
    +                       addr = url.Path
    +                       addrToUse = uri
    +               } else {
    +                       opts = append(opts,
    +                               grpc.WithContextDialer(
    +                                       func(ctx context.Context, addr string) (net.Conn, error) {
    +                                               return (&net.Dialer{}).DialContext(ctx, url.Scheme, addr)
    +                                       },
    +                               ))
                    }
                    log.WithField(
                            "address",
                            fmt.Sprintf("%s://%s", url.Scheme, addr),
                    ).Infof("Connecting to key service")
    -               conn, err := grpc.NewClient(addr, opts...)
    +               conn, err := grpc.NewClient(addrToUse, opts...)
                    if err != nil {
                            log.Fatalf("failed to listen: %v", err)
                    }

    solves the problem. I cannot really claim that I understand whether it has unwanted side-effects, though...

    @getsops/maintainers does anyone of you have a better idea about this?

  3. matheuscscp commented on Aug 10, 2025

    @matheuscscp
    Contributor

    @felixfontein This fix is completely safe, the *net.Dialer that will be used for unix sockets without the grpc.WithContextDialer() option is the following:

    https://github.com/grpc/grpc-go/blob/57b69b47a2da17098c387c8c64c4aab5295cbd3e/internal/tcp_keepalive_unix.go#L30-L54

    // NetDialerWithTCPKeepalive returns a net.Dialer that enables TCP keepalives on
    // the underlying connection with OS default values for keepalive parameters.
    //
    // TODO: Once https://github.com/golang/go/issues/62254 lands, and the
    // appropriate Go version becomes less than our least supported Go version, we
    // should look into using the new API to make things more straightforward.
    func NetDialerWithTCPKeepalive() *net.Dialer {
    	return &net.Dialer{
    		// Setting a negative value here prevents the Go stdlib from overriding
    		// the values of TCP keepalive time and interval. It also prevents the
    		// Go stdlib from enabling TCP keepalives by default.
    		KeepAlive: time.Duration(-1),
    		// This method is called after the underlying network socket is created,
    		// but before dialing the socket (or calling its connect() method). The
    		// combination of unconditionally enabling TCP keepalives here, and
    		// disabling the overriding of TCP keepalive parameters by setting the
    		// KeepAlive field to a negative value above, results in OS defaults for
    		// the TCP keepalive interval and time parameters.
    		Control: func(_, _ string, c syscall.RawConn) error {
    			return c.Control(func(fd uintptr) {
    				unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_KEEPALIVE, 1)
    			})
    		},
    	}
    }

    For unix sockets, this *net.Dialer is functionally equivalent to the one being used before (i.e. &net.Dialer{}), because there's no such thing as TCP keepalive in unix sockets.

    Note: We could also remove the grpc.WithContextDialer() option entirely, as this default *net.Dialer chosen by grpc is probably fine for TCP. However, to fully maintain the current TCP behavior I would go with your fix as-is. PR: #1910

  4. RussellAult commented on Sep 9, 2025

    @RussellAult
    Contributor

    Thanks for fixing this! Is there any chance of getting a release with the fix soon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions