Repository navigation
Closed
Labels
Description
I want run run a sops key service and provide a unix socket via bind mount to containers. This appears broken in 3.10.2 even when running without any bind mounts at all, but worked with 3.9.4.
Setup
sops keyservice --verbose --network unix --address $XDG_RUNTIME_DIR/sops/keyservice
NB: version of the server does not seem to make a difference
sops 3.10.2, broken
.../sops-v3.10.2/sops --verbose --enable-local-keyservice=false --keyservice 'unix:///run/user/1000/sops/keyservice' -d <omitteomittedd>.sops
[CMD] INFO[0000] Connecting to key service address="unix:///run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] original dial target is: "/run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"dns", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"//run/user/1000/sops/keyservice", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel authority set to "%2Frun%2Fuser%2F1000%2Fsops%2Fkeyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel exiting idle mode
2025/04/23 00:00:00 INFO: [core] [Channel #1]Resolver state updated: {
"Addresses": null,
"Endpoints": [],
"ServiceConfig": null,
"Attributes": null
} ()
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel switches to new LB policy "pick_first"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to TRANSIENT_FAILURE
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
<snip>
There is no activity visible via strace on the server. No connection is attempted.
sops 3.9.4, working
.../sops-v3.9.4/sops --verbose --enable-local-keyservice=false --keyservice 'unix:///run/user/1000/sops/keyservice' -d <omitted>.sops
[CMD] INFO[0000] Connecting to key service address="unix:///run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] original dial target is: "/run/user/1000/sops/keyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"//run/user/1000/sops/keyservice", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel authority set to "%2Frun%2Fuser%2F1000%2Fsops%2Fkeyservice"
2025/04/23 00:00:00 INFO: [core] [Channel #1]Resolver state updated: {
"Addresses": [
{
"Addr": "/run/user/1000/sops/keyservice",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "/run/user/1000/sops/keyservice",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel switches to new LB policy "pick_first"
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel created
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to CONNECTING
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel exiting idle mode
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to CONNECTING
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel picks a new address "/run/user/1000/sops/keyservice" to connect
2025/04/23 00:00:00 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to READY
2025/04/23 00:00:00 INFO: [core] [Channel #1]Channel Connectivity change to READY
[SOPS] INFO[0000] Data key recovered successfully
[SOPS] DEBU[0000] Decrypting tree
<snip>
Investigation
Possible change: 060f7a4 (Stop using deprecated grpc.Dial(), use grpc.NewClient() instead., 2025-03-02)
Reacted by Sebastian Gaiser and RussellAult
Activity
chirayuk commentedon Jul 22, 2025
I'm experiencing this issue as well on v3.10.2 (and v3.10.1). Unix socket fails (tcp works fine). No logs on the server process.
felixfontein commentedon Jul 23, 2025
I can confirm that this is caused by 060f7a4.
When playing around a bit, it seems that
solves the problem. I cannot really claim that I understand whether it has unwanted side-effects, though...
@getsops/maintainers does anyone of you have a better idea about this?
matheuscscp commentedon Aug 10, 2025
@felixfontein This fix is completely safe, the
*net.Dialerthat will be used for unix sockets without thegrpc.WithContextDialer()option is the following:https://github.com/grpc/grpc-go/blob/57b69b47a2da17098c387c8c64c4aab5295cbd3e/internal/tcp_keepalive_unix.go#L30-L54
For unix sockets, this
*net.Dialeris functionally equivalent to the one being used before (i.e.&net.Dialer{}), because there's no such thing as TCP keepalive in unix sockets.Note: We could also remove the
grpc.WithContextDialer()option entirely, as this default*net.Dialerchosen bygrpcis probably fine for TCP. However, to fully maintain the current TCP behavior I would go with your fix as-is. PR: #1910RussellAult commentedon Sep 9, 2025
Thanks for fixing this! Is there any chance of getting a release with the fix soon?