GitHub Issue - SDC data-server Bug

Repository: https://github.com/sdcio/data-server/issues

Title

gNMI sync fails with Arista cEOS - "invalid message type: *gnmi.SubscribeRequest"

Labels

bug, gnmi, arista, data-server

Issue Body

Description

SDC data-server fails to sync with Arista cEOS using gNMI Subscribe Request. The error occurs with both 4.33.0F and 4.34.1F versions, and with both JSON_IETF and JSON encodings.

RunningConfig remains empty despite Target showing READY=True.

Error Message

{
  "time": "2026-01-08T17:18:54Z",
  "level": "ERROR",
  "msg": "failed to sync target",
  "datastore-name": "sdc.spine-1",
  "logger": "datastore/sync/Sync",
  "err": "option DataType: invalid message type: *gnmi.SubscribeRequest",
  "mode": "once"
}

The error repeats every 2-3 seconds, indicating continuous retry attempts.

Environment

SDC Version:

• config-server: v0.0.54
• data-server: v0.0.66
• schema-server: integrated

Target Device:

• Vendor: Arista Networks
• OS: cEOS (containerized EOS)
• Version 1: 4.34.1F (tested)
• Version 2: 4.33.0F (tested)
• Platform: x86_64-kvm_x86_64-r0
• Image: ceos:4.33.0F and ceos:4.34.1F

Deployment:

• Kubernetes: KinD v1.32.0
• Containerlab: v0.71.1
• Network: Docker bridge (172.20.20.0/24)

Configuration

TargetConnectionProfile:

apiVersion: inv.sdcio.dev/v1alpha1
kind: TargetConnectionProfile
metadata:
  name: gnmi-eos
  namespace: sdc
spec:
  protocol: gnmi
  port: 6030
  encoding: JSON_IETF
  insecure: true
  skipVerify: true
  timeout: 30s
  connectRetry: 30s
  commitCandidate: candidate
  includeNS: false
  operationWithNS: false
  preferredNetconfVersion: "1.0"
  useOperationRemove: false

TargetSyncProfile:

apiVersion: inv.sdcio.dev/v1alpha1
kind: TargetSyncProfile
metadata:
  name: gnmi-sync
  namespace: sdc
spec:
  sync:
  - mode: once
    paths:
    - "/"
    protocol: gnmi
    port: 6030
  buffer: 0
  workers: 2
  validate: true

Target Status:

$ kubectl get target spine-1 -n sdc
NAME      READY   REASON   PROVIDER               VERSION   ADDRESS
spine-1   True             eos.arista.sdcio.dev   4.33.0F   172.20.20.11

RunningConfig Status:

$ kubectl get runningconfig spine-1 -n sdc -o jsonpath='{.status.value}'
{}  # ← Empty

Evidence: gnmic Native Test (CRITICAL)

To isolate whether the issue was in SDC or Arista, we tested gNMI Subscribe directly using gnmic v0.37.0 (OpenConfig official tool).

Test Setup:

• Tool: gnmic v0.37.0
• Target: Same Arista cEOS switch (172.20.20.11:6030)
• Credentials: admin/admin
• Subscribe mode: once (same as SDC)
• Subscribe path: "/" (same as SDC)

Results:

Commands Executed:

# JSON_IETF - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json_ietf \
  subscribe --path "/" --mode once

# JSON - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json \
  subscribe --path "/" --mode once

# PROTO - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding proto \
  subscribe --path "/" --mode once

Sample Data Received (truncated):

{
  "source": "172.20.20.11:6030",
  "subscription-name": "default-1767893168",
  "timestamp": 1767892423366941952,
  "time": "2026-01-08T17:13:43.366941952Z",
  "prefix": "system/messages",
  "updates": [
    {
      "Path": "config/severity",
      "values": {
        "config/severity": "DEBUG"
      }
    }
  ]
}

Paths Successfully Retrieved:

• system/messages
• system/ssh-server/state/counters
• system/console/state/counters
• arista/redundancy/management-active
• system/aaa/authentication/users/user[username=admin]/state
• Hundreds of other paths (truncated for brevity)

Conclusion

This proves the issue is in SDC data-server, NOT in Arista EOS:

✅ Arista cEOS gNMI implementation works perfectly
✅ All encodings supported (JSON_IETF, JSON, PROTO)
✅ Subscribe mode "once" works correctly
✅ Path "/" returns complete data

❌ SDC data-server constructs gNMI SubscribeRequest incorrectly
❌ Error mentions "option DataType: invalid message type"
❌ gnmic does NOT send this "DataType" option and works fine

Hypothesis

The error message "option DataType: invalid message type: *gnmi.SubscribeRequest" suggests that:

1. SDC is adding a custom option called "DataType" to the gNMI SubscribeRequest
2. Arista EOS validates this option and rejects it as invalid
3. gnmic does NOT send this option (or sends it correctly) and succeeds

gNMI Specification Reference:

According to the gNMI proto spec, SubscribeRequest should use Encoding field in SubscriptionList, not a custom "DataType" option:

message SubscriptionList {
  repeated Subscription subscription = 1;
  // ...
  Encoding encoding = 4;  // ← Standard field
}

Steps to Reproduce

1. Deploy Arista cEOS 4.33.0F or 4.34.1F using containerlab
2. Enable gNMI on port 6030 with admin/admin credentials
3. Install SDC v0.0.54+ in Kubernetes cluster
4. Create TargetConnectionProfile with encoding=JSON_IETF
5. Create TargetSyncProfile with mode=once, paths=["/"]
6. Create Target pointing to Arista switch
7. Observe logs: kubectl logs -n sdc config-server-XXX -c data-server

Expected Behavior:

• RunningConfig should be populated with switch state
• No errors in data-server logs

Actual Behavior:

• RunningConfig remains empty: {}
• Error repeats every 2-3s: "option DataType: invalid message type"

Workaround

Use gnmic directly for testing/validation:

gnmic -a <switch-ip>:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json_ietf \
  subscribe --path "/" --mode once

For production, consider using Ansible with arista.eos collection until this bug is fixed.

Impact

Severity: HIGH

• ❌ Blocks ConfigSets (Phase 3) - cannot apply declarative configurations
• ❌ Blocks GitOps workflows
• ❌ RunningConfig always empty
• ❌ Affects ALL Arista cEOS versions tested (4.33.0F, 4.34.1F)
• ❌ Affects multiple encodings (JSON_IETF, JSON)

Affected Users:

• Anyone using SDC with Arista cEOS
• Anyone using SDC with Arista hardware switches (potentially)

Additional Information

Tests Performed:

1. ✅ Encoding change (JSON_IETF → JSON): Did NOT resolve
2. ✅ EOS version downgrade (4.34.1F → 4.33.0F): Did NOT resolve
3. ✅ Schema update (using official 4.33.0F schema): Did NOT resolve
4. ✅ gnmic native test: RESOLVED - proves bug is in SDC

Time Invested:

• ~12 hours of systematic troubleshooting
• Multiple hypotheses tested and ruled out
• Root cause definitively isolated to SDC data-server

Documentation:

• Complete troubleshooting process documented
• All configurations tested and validated
• Evidence collected for easy reproduction

Logs

SDC data-server Error (repeating):

{"time":"2026-01-08T17:18:54Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}
{"time":"2026-01-08T17:18:57Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}
{"time":"2026-01-08T17:19:00Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}

gnmic Success (sample):

$ gnmic -a 172.20.20.11:6030 -u admin -p admin --insecure \
    --encoding json_ietf subscribe --path "/" --mode once

{
  "source": "172.20.20.11:6030",
  "subscription-name": "default-1767893168",
  "timestamp": 1767892423366941952,
  "time": "2026-01-08T17:13:43.366941952Z",
  "prefix": "system/messages",
  "updates": [
    {
      "Path": "config/severity",
      "values": {
        "config/severity": "DEBUG"
      }
    }
  ]
}
[... hundreds more paths successfully received ...]

Suggested Fix

Investigation needed in data-server code:

1. Search for where gNMI SubscribeRequest is constructed
2. Check if a custom "DataType" option is being added
3. Compare with gnmic implementation (which works)
4. Remove or fix the "DataType" option to match gNMI spec

Code locations to check:

# In data-server repository
grep -r "DataType" --include="*.go"
grep -r "SubscribeRequest" --include="*.go"
grep -r "SubscriptionList" --include="*.go"

Related Issues

• (Search for similar Arista/gNMI issues in sdcio repos)

References

• gNMI Specification: https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto
• gnmic Tool: https://github.com/openconfig/gnmic
• Arista YANG Models: https://github.com/aristanetworks/yang
• SDC Documentation: https://docs.sdcio.dev

Attachments

(When creating the actual issue, attach these files):

1. data-server-logs.txt - Complete logs showing repeated errors
2. gnmic-success-output.txt - Full gnmic output showing success
3. target-config.yaml - Complete Target/Profile configurations
4. target-status.yaml - kubectl get target -o yaml output

Contact

Available for:

• Providing additional logs/configs
• Testing patches
• Validating fixes
• Contributing PR if pointed to the right code location

Thank you for maintaining SDC! This is a great project and we're eager to see Arista support working properly.