Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gNMI - TLS handshake failure on Ciena devices #16476

Closed
whizkidTRW opened this issue Feb 4, 2025 · 4 comments · Fixed by #16507
Closed

gNMI - TLS handshake failure on Ciena devices #16476

whizkidTRW opened this issue Feb 4, 2025 · 4 comments · Fixed by #16507
Assignees
@srebhan
Labels
bug unexpected problem or unintended behavior

Comments

@whizkidTRW
Copy link

whizkidTRW commented Feb 4, 2025 •
edited
Loading

Relevant telegraf.conf

[[inputs.gnmi]]
  interval = "5m"
  alias = "ciena-gnmi"
  addresses = [ "XX.XX.XX.XX:6702" ]
      
  username = "XXXXXXXXXXXXXX"
  password = "XXXXXXXXXXXXXX"
      
  encoding = "proto"
  redial = "10s"
  tls_enable = true
  insecure_skip_verify = true
  tls_ca = "/etc/telegraf/ciena-ca.cert.pem"
  tls_cert = "/etc/telegraf/ciena-client.cert.pem"
  tls_key = "/etc/telegraf/ciena-client.key.pem"
  name_override = "saos10xgnmi"
  updates_only = true

  fieldpass = ["path","source","name", "in_crc_error_pkts", "in_discards", "in_errors", "in_octets", "out_errors", "out_octets"]
  tagexclude = ["path","name"]

  [[inputs.gnmi.subscription]]
     name = "ifcounters"
     origin = "Ciena"
     path = "/oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters"
     subscription_mode = "sample"
     sample_interval = "30s"

Logs from Telegraf

Broken starting with the 1.29.2 release (through latest):
---------------------------------------------------------
telegraf  | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf  | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf  | 2025-02-04T14:40:45Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf  | 2025-02-04T14:40:45Z I! Starting Telegraf 1.29.2 brought to you by InfluxData the makers of InfluxDB
telegraf  | 2025-02-04T14:40:45Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf  | 2025-02-04T14:40:45Z I! Loaded inputs: gnmi
telegraf  | 2025-02-04T14:40:45Z I! Loaded aggregators: 
telegraf  | 2025-02-04T14:40:45Z I! Loaded processors: converter rename strings
telegraf  | 2025-02-04T14:40:45Z I! Loaded secretstores: 
telegraf  | 2025-02-04T14:40:45Z W! Outputs are not used in testing mode!
telegraf  | 2025-02-04T14:40:45Z I! Tags enabled: host=10.5.200.224
telegraf  | 2025-02-04T14:40:45Z D! [agent] Initializing plugins
telegraf  | 2025-02-04T14:40:45Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf  | 2025-02-04T14:40:45Z D! [agent] Starting service inputs
telegraf  | 2025-02-04T14:40:45Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
telegraf  | 2025-02-04T14:40:55Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
 ✔ Container telegraf  Stopped                                                                                                                                                                                                



Exact same config works fine up to 1.29.1 release:
--------------------------------------------------
telegraf  | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf  | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf  | 2025-02-04T14:41:15Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf  | 2025-02-04T14:41:15Z I! Starting Telegraf 1.29.1 brought to you by InfluxData the makers of InfluxDB
telegraf  | 2025-02-04T14:41:15Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf  | 2025-02-04T14:41:15Z I! Loaded inputs: gnmi
telegraf  | 2025-02-04T14:41:15Z I! Loaded aggregators: 
telegraf  | 2025-02-04T14:41:15Z I! Loaded processors: converter rename strings
telegraf  | 2025-02-04T14:41:15Z I! Loaded secretstores: 
telegraf  | 2025-02-04T14:41:15Z W! Outputs are not used in testing mode!
telegraf  | 2025-02-04T14:41:15Z I! Tags enabled: host=10.5.200.224
telegraf  | 2025-02-04T14:41:15Z D! [agent] Initializing plugins
telegraf  | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf  | 2025-02-04T14:41:15Z D! [agent] Starting service inputs
telegraf  | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Connection to gNMI device 10.255.32.14:6702 established
telegraf  | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=1 ifHCInOctets=94533575503546i,ifHCOutOctets=873939187818389i,ifInCrcErrors=0i,ifInDiscards=236i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000
telegraf  | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=2 ifHCInOctets=94569510275635i,ifHCOutOctets=874902982673682i,ifInCrcErrors=0i,ifInDiscards=0i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000

. . . (output trimmed fo clarity)

Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
 ✔ Container telegraf  Stopped

System info

Telegraf 1.29.2+, Docker 4.37.2, MacOS 15.2 (m4-Max)

Docker

services:
  telegraf:
    image: telegraf:1.29.2-alpine
    container_name: telegraf
    restart: no
    command: telegraf --debug --test-wait 45
    volumes:
      - /etc/snmp:/etc/snmp:ro
      - ./mibs:/usr/share/snmp/mibs:rw
      - ./telegraf/etc:/etc/telegraf:rw
    ports: 
      - '8125:8125'
    logging:
      options:
        max-size: "1m"
        max-file: "5"

Steps to reproduce

  1. Start with Telegraf 1.29.2
  2. Subscribe to a Ciena device
  3. TLS Authentication fails (certs are valid / vendor supplied, don't expire until 2050)
  4. Revert to Telegraf 1.29.1, connection works fine

Expected behavior

TLS handshake is expected to still work with known good config from 1.29.1 to subsequent versions

Actual behavior

TLS handshake breaks starting in 1.29.2

Additional info

Confirmed to be working with my Cisco IOS-XR devices, so this problem is unique to Ciena.
@whizkidTRW whizkidTRW added the bug unexpected problem or unintended behavior label Feb 4, 2025
@srebhan
Copy link
Member

srebhan commented Feb 10, 2025

@whizkidTRW golang started to disable insecure TLS cipher suites by default and this is very likely the culprit causing the error. Could you please try to set tls_cipher_suites = ["all"] in the inputs.gnmi section of your config and check if this fixes the issue!?

@srebhan srebhan self-assigned this Feb 10, 2025
@srebhan srebhan added the waiting for response waiting for response from contributor label Feb 10, 2025
@whizkidTRW
Copy link
Author

whizkidTRW commented Feb 10, 2025 •
edited
Loading

Looks like that fixed the auth issue!
But now I have a new "No Measurement alias for gNMI path" error for all measurements.

telegraf  | 2025-02-10T16:56:19Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf  | 2025-02-10T16:56:19Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf  | 2025-02-10T16:56:19Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 1.40.0: use 'fieldinclude' instead
telegraf  | 2025-02-10T16:56:19Z I! Starting Telegraf 1.33.1 brought to you by InfluxData the makers of InfluxDB
telegraf  | 2025-02-10T16:56:19Z I! Available plugins: 236 inputs, 9 aggregators, 33 processors, 26 parsers, 63 outputs, 6 secret-stores
telegraf  | 2025-02-10T16:56:19Z I! Loaded inputs: gnmi
telegraf  | 2025-02-10T16:56:19Z I! Loaded aggregators:
telegraf  | 2025-02-10T16:56:19Z I! Loaded processors: converter rename strings
telegraf  | 2025-02-10T16:56:19Z I! Loaded secretstores:
telegraf  | 2025-02-10T16:56:19Z W! Outputs are not used in testing mode!
telegraf  | 2025-02-10T16:56:19Z I! Tags enabled: host=10.5.200.224
telegraf  | 2025-02-10T16:56:19Z D! [agent] Initializing plugins
telegraf  | 2025-02-10T16:56:19Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/interface/state/counters:ifcounters]
telegraf  | 2025-02-10T16:56:19Z D! [agent] Starting service inputs
telegraf  | 2025-02-10T16:56:19Z D! [inputs.gnmi::ciena-gnmi] Connection to gNMI device 10.255.32.14:6702 established
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-1024-to-1518-octet-pkts
telegraf  | 2025-02-10T16:56:28Z W! [inputs.gnmi::ciena-gnmi] Got empty metric-name for response (field "Ciena:/interfaces/interface/state/counters/in-1024-to-1518-octet-pkts"), usually
telegraf  | indicating configuration issues as the response cannot be related to any
telegraf  | subscription.Please open an issue on https://github.com/influxdata/telegraf
telegraf  | including your device model and the following response data:
telegraf  | {"Update":{"timestamp":1739206587847000000,"prefix":{"origin":"Ciena","elem":[{"name":"oc-if:interfaces"},{"name":"oc-if:interface"},{"name":"oc-if:state"},{"name":"oc-if:counters"}]},"update":[{"path":{"origin":"Ciena","elem":[{"name":"in-1024-to-1518-octet-pkts"}]},"val":{"Value":{"UintVal":15680405047}}},{"path":{"origin":"Ciena","elem":[{"name":"in-128-to-255-octet-pkts"}]},"val":{"Value":{"UintVal":12809649942}}},{"path":{"origin":"Ciena","elem":[{"name":"in-1519-to-2047-octet-pkts"}]},"val":{"Value":{"UintVal":35815850565}}},{"path":{"origin":"Ciena","elem":[{"name":"in-2048-to-4095-octet-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-256-to-511-octet-pkts"}]},"val":{"Value":{"UintVal":5257910993}}},{"path":{"origin":"Ciena","elem":[{"name":"in-4096-to-9216-octet-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-512-to-1023-octet-pkts"}]},"val":{"Value":{"UintVal":6139561818}}},{"path":{"origin":"Ciena","elem":[{"name":"in-64-octet-pkts"}]},"val":{"Value":{"UintVal":4}}},{"path":{"origin":"Ciena","elem":[{"name":"in-65-to-127-octet-pkts"}]},"val":{"Value":{"UintVal":146456592549}}},{"path":{"origin":"Ciena","elem":[{"name":"in-broadcast-pkts"}]},"val":{"Value":{"UintVal":167166}}},{"path":{"origin":"Ciena","elem":[{"name":"in-crc-error-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-discards"}]},"val":{"Value":{"UintVal":236}}},{"path":{"origin":"Ciena","elem":[{"name":"in-discards-octets"}]},"val":{"Value":{"UintVal":31492}}},{"path":{"origin":"Ciena","elem":[{"name":"in-dropped-octets"}]},"val":{"Value":{"UintVal":31492}}},{"path":{"origin":"Ciena","elem":[{"name":"in-dropped-pkts"}]},"val":{"Value":{"UintVal":236}}},{"path":{"origin":"Ciena","elem":[{"name":"in-errors"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-jabber-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-multicast-pkts"}]},"val":{"Value":{"UintVal":76815719}}},{"path":{"origin":"Ciena","elem":[{"name":"in-octets"}]},"val":{"Value":{"UintVal":95890972327359}}},{"path":{"origin":"Ciena","elem":[{"name":"in-oversize-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-pkts"}]},"val":{"Value":{"UintVal":222159970919}}},{"path":{"origin":"Ciena","elem":[{"name":"in-undersize-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"in-unicast-pkts"}]},"val":{"Value":{"UintVal":222082988034}}},{"path":{"origin":"Ciena","elem":[{"name":"last-clear"}]},"val":{"Value":{"UintVal":1679547185677412529}}},{"path":{"origin":"Ciena","elem":[{"name":"link-flap-events"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"name"}]},"val":{"Value":{"StringVal":"\"1\""}}},{"path":{"origin":"Ciena","elem":[{"name":"out-1519-to-2047-octet-pkts"}]},"val":{"Value":{"UintVal":211382493634}}},{"path":{"origin":"Ciena","elem":[{"name":"out-2048-to-4095-octet-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"out-4096-to-9216-octet-pkts"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"out-broadcast-pkts"}]},"val":{"Value":{"UintVal":2609579674}}},{"path":{"origin":"Ciena","elem":[{"name":"out-errors"}]},"val":{"Value":{"UintVal":0}}},{"path":{"origin":"Ciena","elem":[{"name":"out-multicast-pkts"}]},"val":{"Value":{"UintVal":332069076}}},{"path":{"origin":"Ciena","elem":[{"name":"out-octets"}]},"val":{"Value":{"UintVal":885293268981054}}},{"path":{"origin":"Ciena","elem":[{"name":"out-pkts"}]},"val":{"Value":{"UintVal":677379680498}}},{"path":{"origin":"Ciena","elem":[{"name":"out-unicast-pkts"}]},"val":{"Value":{"UintVal":674438031748}}}]}}
telegraf  | This message is only printed once.
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-128-to-255-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-1519-to-2047-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-2048-to-4095-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-256-to-511-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-4096-to-9216-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-512-to-1023-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-64-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-65-to-127-octet-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-broadcast-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-crc-error-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-discards
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-discards-octets
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-dropped-octets
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-dropped-pkts
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-errors
telegraf  | 2025-02-10T16:56:28Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/in-jabber-pkts

. . . output clipped for clarity . . . 

telegraf  | 2025-02-10T16:56:58Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/name
telegraf  | 2025-02-10T16:56:58Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/out-1519-to-2047-octet-pkts
telegraf  | 2025-02-10T16:56:58Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/out-2048-to-4095-octet-pkts
telegraf  | 2025-02-10T16:56:58Z D! [inputs.gnmi::ciena-gnmi] No measurement alias for gNMI path: Ciena:/interfaces/interface/state/counters/out-4096-to-9216-octet-pkts
telegraf exited with code 0

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Feb 10, 2025
@whizkidTRW
Copy link
Author

And, just to confirm, this is the proper sensor name. It's been working for 2+ years, only change is the Telegraf version.

XXXX-XX-RT-CN-01-02> show telemetry sensors available
+----------------------------------------------------------------------------------------- TELEMETRY REGISTERED SENSORS  -----------------------------------------------------------------------------------------+
| Index | Name                | Value                                                                                                                                                                             |
+-------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|       | Registered Sensors: |                                                                                                                                                                                   |
| 1     |   Sensor Path Name  | /pon-ctrl:pon-ctrl-state/pon-ctrl:onus/pon-ctrl:onu/pon-ctrl:datalinks/pon-ctrl:datalink-authentication/pon-ctrl:statistics                                                       |
|       |   Sensor Sub Mode   | sample                                                                                                                                                                            |
|       |   Sensor ID         | 394708392011360203                                                                                                                                                                |
+-------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2     |   Sensor Path Name  | /ciena-sys-tmet:system-state/memory                                                                                                                                               |
|       |   Sensor Sub Mode   | sample                                                                                                                                                                            |
|       |   Sensor ID         | 537827677912849737                                                                                                                                                                |
+-------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

. . . output clipped for clarity . . .

+-------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 11    |   Sensor Path Name  | /oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters                                                                                                                      |
|       |   Sensor Sub Mode   | sample                                                                                                                                                                            |
|       |   Sensor ID         | 3071242026345933326

@srebhan srebhan mentioned this issue Feb 11, 2025
Merged
1 task
@srebhan
Copy link
Member

srebhan commented Feb 11, 2025

@whizkidTRW could you please test the binary in PR #16507, available as soon as CI finished the tests, and let me know if this fixes the issue!? You need to set enforce_first_namespace_as_origin = false in the inputs.gnmi section for the test.

@srebhan srebhan added waiting for response waiting for response from contributor and removed waiting for response waiting for response from contributor labels Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srebhan srebhan

Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(inputs.gnmi): Allow to disable using first namespace as origin srebhan/telegraf
2 participants
@whizkidTRW @srebhan