For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/integrations/portworx.md. A documentation index is available at /llms.txt.

Portworx

Supported OS Linux

Integration version1.1.1

To find out if this integration is available in your organization, see your Datadog Integrations page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email [email protected].

Overview

Get metrics from Portworx service in real time to:

  • Monitor health and performance of your Portworx cluster
  • Track disk usage, latency, and throughput for Portworx volumes
  • Set up Autopilot rules

Setup

Installation

Create the Datadog credentials Secret

Create a Kubernetes Secret containing your Datadog API and application keys:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-credentials
  namespace: datadog-ns
type: Opaque
data:
  # Base64-encoded Datadog API Key
  api-key: <base64-encoded-api-key>
  # Base64-encoded Datadog Application Key
  app-key: <base64-encoded-app-key>
  • Replace the namespace, secret name, and encoded key values as required.
  • Use an application key with metrics read permission for app-key.

Configure the Datadog Agent to export Portworx metrics

Create a Datadog Agent values file (for example, datadog_config.yaml):

datadog:
  site: "datadoghq.com"
  clusterName: "your-cluster-name"
  apiKeyExistingSecret: "datadog-credentials"
  kubelet:
    tlsVerify: false
  clusterChecks:
    enabled: true
  orchestratorExplorer:
    enabled: true
clusterAgent:
  enabled: true
  admissionController:
    enabled: true
    mutateUnlabelled: false

Use your own site, clusterName, and apiKeyExistingSecret. The Agent scrapes Portworx metrics and sends them to Datadog.

Install the Datadog Agent with Helm:

helm repo add datadog https://helm.datadoghq.com
helm upgrade --install datadog-agent datadog/datadog -f ./datadog_config.yaml

Configuration

Annotate Portworx, Stork, and Autopilot pods

Configure Datadog Autodiscovery annotations so the Agent knows which endpoints to scrape. The typical Prometheus-style service endpoints are:

ComponentEndpointMetrics filter
Portworx APIhttp://%%host%%:17001/metricspx_*
Storkhttp://%%host%%:9091/metricsstork_*
Autopilothttp://%%host%%:9628/metricsautopilot_*

Note: Portworx has many metrics. For most use cases, Portworx API metrics alone are sufficient.

Apply a ComponentK8sConfig Custom Resource (CR) to add the annotations:

apiVersion: core.libopenstorage.org/v1
kind: ComponentK8sConfig
metadata:
  name: datadog-components-config
  namespace: portworx
spec:
  components:
  - componentNames:
    - Portworx API
    workloadConfigs:
    - workloadNames:
      - portworx-api
      annotations:
        ad.datadoghq.com/portworx-api.checks: |-
          {
            "openmetrics": {
              "instances": [
                {
                  "openmetrics_endpoint": "http://%%host%%:17001/metrics",
                  "namespace": "portworx",
                  "metrics": ["px_*"]
                }
              ]
            }
          }
  - componentNames:
    - Stork
    workloadConfigs:
    - workloadNames:
      - stork
      annotations:
        ad.datadoghq.com/stork.checks: |-
          {
            "openmetrics": {
              "instances": [
                {
                  "openmetrics_endpoint": "http://%%host%%:9091/metrics",
                  "namespace": "portworx",
                  "metrics": ["stork_*"]
                }
              ]
            }
          }
  - componentNames:
    - Autopilot
    workloadConfigs:
    - workloadNames:
      - autopilot
      annotations:
        ad.datadoghq.com/autopilot.checks: |-
          {
            "openmetrics": {
              "instances": [
                {
                  "openmetrics_endpoint": "http://%%host%%:9628/metrics",
                  "namespace": "portworx",
                  "metrics": ["autopilot_*"]
                }
              ]
            }
          }

At this point, the Datadog Agent is configured to scrape Portworx metrics and the Portworx, Stork, and Autopilot pods are annotated for Autodiscovery.

To configure rules, see the Autopilot documentation.

Validation

Run the following command to confirm the Agent is scraping Portworx metrics:

kubectl exec <datadog-agent-pod> -n <datadog-namespace> -- agent status

Look for openmetrics instances under the Checks section with portworx, stork, and autopilot.

Compatibility

This integration is compatible with Portworx 1.4.0 and later versions.

Data Collected

Metrics

portworx.px_backup_stats_backup_status
(gauge)
Status for this backup (0=InProgress,1=Done,2=Failed)
portworx.px_backup_stats_backup_size
(gauge)
Size in bytes for this backup
Shown as byte
portworx.px_backup_stats_backup_duration_seconds
(gauge)
Duration in seconds for this backup
Shown as second
portworx.px_backup_stats_backup_uploaded_bytes_mbs
(gauge)
Delta bytes uploaded in MB/s from last interval for this backup
Shown as byte
portworx.px_backup_stats_download_size_bytes
(gauge)
Size of downloaded bytes during backup/restore from cloud
Shown as byte
portworx.px_backup_stats_upload_size_bytes
(gauge)
Size of uploaded bytes during backup/restore from cloud
Shown as byte
portworx.px_backup_stats_get_apis_invoked
(gauge)
Number of times GET API was invoked
portworx.px_backup_stats_put_apis_invoked
(gauge)
Number of times PUT API was invoked
portworx.px_cluster_cpu_percent
(gauge)
Percentage of CPU used
Shown as percent
portworx.px_cluster_memory_utilized_percent
(gauge)
Percentage of memory utilization
Shown as percent
portworx.px_cluster_disk_total_bytes
(gauge)
Total storage space in bytes for this node
Shown as byte
portworx.px_cluster_disk_available_bytes
(gauge)
Available storage space in bytes for this node
Shown as byte
portworx.px_cluster_disk_utilized_bytes
(gauge)
Utilized storage space in bytes for this node
Shown as byte
portworx.px_cluster_pendingio
(gauge)
Number of read and write operations currently in progress for this node
portworx.px_cluster_trashcan_volumes
(gauge)
Number of volumes currently in the trashcan
portworx.px_cluster_status_cluster_size
(gauge)
Node count for your Portworx cluster. Deprecated.
portworx.px_cluster_status_size
(gauge)
Node count for your Portworx cluster
portworx.px_cluster_status_cluster_quorum
(gauge)
Indicates if the cluster is in quorum. Deprecated.
portworx.px_cluster_status_quorum
(gauge)
Indicates if the cluster is in quorum
portworx.px_cluster_status_nodes_online
(gauge)
Number of online nodes in the cluster (includes storage and storageless)
portworx.px_cluster_status_nodes_offline
(gauge)
Number of offline nodes in the cluster (includes storage and storageless)
portworx.px_cluster_status_nodes_storage_down
(gauge)
Number of nodes where the storage is full or down
portworx.px_cluster_status_storage_nodes_online
(gauge)
Number of storage nodes that are online
portworx.px_cluster_status_storage_nodes_offline
(gauge)
Number of storage nodes that are offline
portworx.px_cluster_status_storage_nodes_decommissioned
(gauge)
Number of storage nodes that are decommissioned
portworx.px_device_delete_delete_after_discard_enabled
(gauge)
This is 1 if NBDD is enabled
portworx.px_device_delete_discard_bytes_total
(count)
Total bytes discarded across discard requests
Shown as byte
portworx.px_device_delete_discard_duration_msec_total
(count)
Total time spent for discard requests
Shown as millisecond
portworx.px_device_delete_discard_errors_total
(count)
Total errors across discard requests
portworx.px_device_delete_discard_operations_total
(count)
Total number of discard requests
portworx.px_device_delete_operations_completed_total
(count)
Total number of delete operations completed
portworx.px_device_delete_operations_inprogress
(gauge)
Current operations in progress
portworx.px_device_delete_operations_queued
(gauge)
Current operations queued
portworx.px_disk_stats_used_bytes
(gauge)
Total storage in bytes for this disk
Shown as byte
portworx.px_disk_stats_interval_seconds
(gauge)
Length of the disk stats sampling interval in seconds
Shown as second
portworx.px_disk_stats_io_seconds
(gauge)
Time spent doing I/O in seconds for this disk
Shown as second
portworx.px_disk_stats_progress_io
(gauge)
I/O operations currently in progress for this disk
portworx.px_disk_stats_disk_read_bytes
(gauge)
Total bytes read for this disk. Deprecated.
Shown as byte
portworx.px_disk_stats_read_bytes
(gauge)
Total bytes read for this disk
Shown as byte
portworx.px_disk_stats_write_bytes_seconds
(gauge)
Total written bytes for this disk. Deprecated.
Shown as byte
portworx.px_disk_stats_written_bytes
(gauge)
Total written bytes for this disk
Shown as byte
portworx.px_disk_stats_read_seconds
(gauge)
Total time spent reading in seconds for this disk
Shown as second
portworx.px_disk_stats_write_seconds
(gauge)
Total time spent writing in seconds for this disk
Shown as second
portworx.px_disk_stats_read_latency_seconds
(gauge)
Average time spent per read operation in seconds for this disk
Shown as second
portworx.px_disk_stats_write_latency
(gauge)
Average time spent per write operation in seconds for this disk. Deprecated.
Shown as second
portworx.px_disk_stats_write_latency_seconds
(gauge)
Average time spent per write operation in seconds for this disk
Shown as second
portworx.px_disk_stats_disk_num_reads
(gauge)
Total number of read operations completed successfully for this disk. Deprecated.
portworx.px_disk_stats_disk_num_writes
(gauge)
Total number of write operations completed successfully for this disk. Deprecated.
portworx.px_disk_stats_num_reads
(gauge)
Total number of read operations completed successfully for this disk
portworx.px_disk_stats_num_writes
(gauge)
Total number of write operations completed successfully for this disk
portworx.px_disk_stats_num_reads_total
(count)
Total number of read operations completed successfully for this disk
portworx.px_disk_stats_num_writes_total
(count)
Total number of write operations completed successfully for this disk
portworx.px_disk_stats_written_bytes_total
(count)
Total bytes written for this disk
Shown as byte
portworx.px_disk_stats_read_bytes_total
(count)
Total bytes read for this disk
Shown as byte
portworx.px_disk_stats_read_seconds_total
(count)
Total time spent reading in seconds for this disk
Shown as second
portworx.px_disk_stats_write_seconds_total
(count)
Total time spent writing in seconds for this disk
Shown as second
portworx.px_fafb_apis_requests_total
(count)
Total number of API requests made to all the configured FA/FB endpoints
portworx.px_fafb_apis_array_requests_total
(count)
Total number of API requests made to the /arrays endpoint.
portworx.px_fafb_apis_login_requests_total
(count)
Total number of API requests made to the /login endpoint.
portworx.px_fafb_apis_controllers_requests_total
(count)
Total number of API requests made to the /controllers endpoint
portworx.px_fafb_apis_alerts_requests_total
(count)
Total number of API requests made to the /alerts endpoint
portworx.px_fafb_apis_ports_requests_total
(count)
Total number of API requests made to the /ports endpoint
portworx.px_fafb_apis_volumes_requests_total
(count)
Total number of API requests made to the /volumes endpoint
portworx.px_fafb_apis_volumesnapshots_requests_total
(count)
Total number of API requests made to the /volume-snapshots endpoint
portworx.px_fafb_apis_hosts_requests_total
(count)
Total number of API requests made to the /hosts endpoint
portworx.px_fafb_apis_connections_requests_total
(count)
Total number of API requests made to the /connections endpoint
portworx.px_fafb_apis_version_requests_total
(count)
Total number of API requests made to the /api_version endpoint
portworx.px_kvdb_get_requests_total
(count)
Total number of get requests for a key
portworx.px_kvdb_snapshot_requests_total
(count)
Total number of kvdb snapshot requests for a list of prefixes
portworx.px_kvdb_put_requests_total
(count)
Total number of put requests for a key
portworx.px_kvdb_create_requests_total
(count)
Total number of create requests for a key
portworx.px_kvdb_update_requests_total
(count)
Total number of update requests for a key
portworx.px_kvdb_enumerate_requests_total
(count)
Total number of enumerate requests for a key
portworx.px_kvdb_delete_requests_total
(count)
Total number of delete requests for a key
portworx.px_kvdb_keys_requests_total
(count)
Total number of list key requests for a prefix
portworx.px_kvdb_cas_requests_total
(count)
Total number of compare and sets for a key
portworx.px_kvdb_cad_requests_total
(count)
Total number of compare and deletes for a key
portworx.px_kvdb_lock_requests_total
(count)
Total number of lock requests for a key
portworx.px_kvdb_unlock_requests_total
(count)
Total number of unlock requests for a key
portworx.px_kvdb_watchkey_requests_total
(count)
Total number of watch requests for a key from a node
portworx.px_kvdb_watchtree_requests_total
(count)
Total number of watch requests for a prefix from a node
portworx.px_kvdb_adduser_requests_total
(count)
Total number of add user requests
portworx.px_kvdb_removeuser_requests_total
(count)
Total number of remove user requests
portworx.px_kvdb_grantuser_access_requests_total
(count)
Total number of grant access requests for a user
portworx.px_kvdb_revokeuser_access_requests_total
(count)
Total number of revoke access requests for a user
portworx.px_kvdb_addmember_requests_total
(count)
Total number of add member requests from a node
portworx.px_kvdb_addlearner_requests_total
(count)
Total number of add learner requests from a node
portworx.px_kvdb_removemember_requests_total
(count)
Total number of remove member requests from a node
portworx.px_kvdb_updatemember_requests_total
(count)
Total number of update member requests from a node
portworx.px_kvdb_listmembers_requests_total
(count)
Total number of list member requests from a node
portworx.px_kvdb_setendpoints_requests_total
(count)
Total number of set endpoint requests from a node
portworx.px_kvdb_getendpoints_requests_total
(count)
Total number of get endpoint requests from a node
portworx.px_kvdb_defragment_requests_total
(count)
Total number of defragment requests from a node
portworx.px_kvdb_watch_latency_seconds
(gauge)
Time taken in seconds between a kvdb put and a corresponding watch update
Shown as second
portworx.px_kvdb_health_state_node_view
(gauge)
This node’s kvdb state (1 = healthy, 2 = not healthy)
portworx.px_kvdb_health_state_cluster_view
(gauge)
This node’s view of other node’s kvdb state (1 = healthy, 2 = not healthy)
portworx.px_network_io_bytessent
(gauge)
Number of bytes sent during this interval
Shown as byte
portworx.px_network_io_received_bytes
(gauge)
Number of bytes received during this interval
Shown as byte
portworx.px_network_io_sent_bytes_total
(count)
Total number of bytes sent
Shown as byte
portworx.px_network_io_received_bytes_total
(count)
Total number of bytes received
Shown as byte
portworx.px_network_io_reservation_waits
(gauge)
Total number of SM waits for reservation
portworx.px_network_io_reservation_wake_ups
(gauge)
Total number of SM wake ups after getting the reservation
portworx.px_nfs_pool_stats_packets_arrived
(gauge)
Counts the number of NFS packets arrived on the socket.
portworx.px_nfs_pool_stats_sockets_enqueued
(gauge)
Counts how many times an NFS transport is enqueued to wait for an nfsd thread to service it, i.e. no nfsd thread was considered available.
portworx.px_nfs_pool_stats_threads_woken
(gauge)
Counts how many times an idle nfsd thread is woken to try to receive some data from an NFS transport.
portworx.px_nfs_pool_stats_threads_timedout
(gauge)
Counts how many times an nfsd thread triggered an idle timeout, i.e. was not woken to handle any incoming network packets for some time.
portworx.px_node_stats_used_mem
(gauge)
Used memory in bytes
Shown as byte
portworx.px_node_stats_cpu_usage
(gauge)
Percent of CPU consumption
Shown as percent
portworx.px_node_stats_busy_threads_percent
(gauge)
Percentage of Busy Threads
Shown as percent
portworx.px_node_stats_thread_pool_high_priority_queue_size
(gauge)
Thread Pool High Priority Queue Size
portworx.px_node_stats_writeback_list_num_dirty_pools
(gauge)
Writeback List Number of Dirty Pools
portworx.px_node_stats_writeback_list_num_pending_log_bufs
(gauge)
Writeback List Number of Pending Log Buffers
portworx.px_node_stats_writeback_list_low_cos_queue_size
(gauge)
Writeback List Low Cos Queue Size
portworx.px_node_stats_writeback_list_high_cos_queue_size
(gauge)
Writeback List High Cos Queue Size
portworx.px_node_stats_sm_run_queue_size
(gauge)
Statemachine Run Queue Size
portworx.px_node_stats_kaio_pending_queue_size
(gauge)
Kaio Pending Queue Size
portworx.px_node_stats_request_scheduler_queue_size
(gauge)
Request Scheduler Queue Size
portworx.px_node_stats_procfs_mem_available_bytes
(gauge)
Available memory in bytes
Shown as byte
portworx.px_node_stats_procfs_mem_dirty_bytes
(gauge)
The total amount of memory waiting to be written back to the disk
Shown as byte
portworx.px_node_stats_procfs_mem_writeback_bytes
(gauge)
The total amount of memory actively being written back to the disk
Shown as byte
portworx.px_node_stats_procfs_mem_total_bytes
(gauge)
Total amount of usable RAM which is physical RAM minus a number of reserved bits and the kernel binary code
Shown as byte
portworx.px_node_stats_procfs_mem_free_bytes
(gauge)
The amount of physical RAM, in bytes, left unused by the system.
Shown as byte
portworx.px_node_stats_procfs_mem_buffers_bytes
(gauge)
The amount, in bytes, of temporary storage for raw disk blocks
Shown as byte
portworx.px_node_stats_procfs_mem_cached_bytes
(gauge)
The amount of physical RAM, in bytes, used as cache memory
Shown as byte
portworx.px_node_stats_procfs_mem_active_anon_bytes
(gauge)
The amount of anonymous and tmpfs/shmem memory, in bytes, that is in active use
Shown as byte
portworx.px_node_stats_procfs_mem_inactive_anon_bytes
(gauge)
The amount of anonymous and tmpfs/shmem memory, in bytes, that is a candidate for eviction
Shown as byte
portworx.px_node_stats_procfs_mem_active_file_bytes
(gauge)
The amount of file cache memory, in bytes, that is in active use
Shown as byte
portworx.px_node_stats_procfs_mem_inactive_file_bytes
(gauge)
The amount of file cache memory, in bytes, that is newly loaded from the disk
Shown as byte
portworx.px_node_stats_procfs_mem_unevictable_bytes
(gauge)
The amount of memory, in bytes, discovered by the pageout code
Shown as byte
portworx.px_node_stats_procfs_mem_mlocked_bytes
(gauge)
The total amount of memory, in bytes, that is not evictable because it is locked into memory by user programs
Shown as byte
portworx.px_node_stats_procfs_mem_anon_pages_bytes
(gauge)
The total amount of memory, in bytes, used by pages that are not backed by files and are mapped into userspace page tables
Shown as byte
portworx.px_node_stats_procfs_mem_mapped_bytes
(gauge)
The memory, in bytes, used for files that have been mmaped
Shown as byte
portworx.px_node_stats_procfs_mem_sh_mem_bytes
(gauge)
The total amount of memory, in bytes, used by shared memory (shmem) and tmpfs
Shown as byte
portworx.px_node_stats_num_skinnysnaps
(gauge)
The number of SkinnySnaps
portworx.px_node_stats_skinnysnaps_num_repls_skips
(gauge)
The number of target replicas skipped for a snapshot due to SkinnySnap
portworx.px_node_stats_skinnysnaps_num_repls_snapshots
(gauge)
The number of target replicas that underwent a snapshot operation due to SkinnySnap
portworx.px_node_stats_relaxed_reclaim_pending
(gauge)
The number of volumes in RelaxedReclaim pending queue
portworx.px_node_stats_relaxed_reclaim_skipped
(gauge)
The number of volume deletes that were skipped from being delayed even when RelaxedReclaim was enabled.
portworx.px_node_stats_relaxed_reclaim_deleted
(gauge)
The number of volume deletes done through RelaxedReclaim
portworx.px_node_stats_num_volumes
(gauge)
The number of volumes hosted on this node
portworx.px_node_status_node_status
(gauge)
Status of this node (https://libopenstorage.github.io/w/master.generated-api/#status)
portworx.px_node_status_license_expiry
(gauge)
Number of days until License (or License lease) expires (<0 means Expired)
portworx.px_none_stats_free_mem
(gauge)
Available memory in bytes
Shown as byte
portworx.px_none_stats_total_mem
(gauge)
Total memory in bytes
Shown as byte
portworx.px_pool_stats_pool_written_bytes
(gauge)
Bytes written since last interval for this pool. Deprecated.
Shown as byte
portworx.px_pool_stats_pool_write_latency_seconds
(gauge)
Average time spent per write operation for this pool. Deprecated.
Shown as second
portworx.px_pool_stats_pool_writethroughput
(gauge)
Average number of bytes written per second for this pool. Deprecated.
Shown as byte
portworx.px_pool_stats_pool_flushed_bytes
(gauge)
Number of flushed bytes since last interval for this pool. Deprecated.
Shown as byte
portworx.px_pool_stats_pool_num_flushes
(gauge)
Number of flush(sync) operations since last interval for this pool. Deprecated.
portworx.px_pool_stats_pool_flushms
(gauge)
Latency for flush for this pool. Deprecated.
Shown as millisecond
portworx.px_pool_stats_pool_provisioned_bytes
(gauge)
Provisioned storage space in bytes for this pool. Deprecated.
Shown as byte
portworx.px_pool_stats_pool_status
(gauge)
Status of this Pool (0=Offline,1=Online). Deprecated.
portworx.px_pool_stats_written_bytes
(gauge)
Bytes written since last interval for this pool
Shown as byte
portworx.px_pool_stats_num_writes
(gauge)
Number of write operations in the last interval for this pool
portworx.px_pool_stats_write_ms
(gauge)
Total time in millisecond spent in writing in the last interval for this pool
Shown as millisecond
portworx.px_pool_stats_write_latency_seconds
(gauge)
Average time spent per write operation for this pool
Shown as second
portworx.px_pool_stats_write_iops
(gauge)
Average number of completed write operations per second for this pool
portworx.px_pool_stats_writethroughput
(gauge)
Average number of bytes written per second for this pool
Shown as byte
portworx.px_pool_stats_flushed_bytes
(gauge)
Number of flushed bytes since last interval for this pool
Shown as byte
portworx.px_pool_stats_num_flushes
(gauge)
Number of flush(sync) operations since last interval for this pool
portworx.px_pool_stats_flushms
(gauge)
Latency for flush for this pool
Shown as millisecond
portworx.px_pool_stats_provisioned_bytes
(gauge)
Provisioned storage space in bytes for this pool
Shown as byte
portworx.px_pool_stats_status
(gauge)
Status of this Pool (0=Offline, 1=Online, 2=Full, 3=NotFound, 4=Maintenance)
portworx.px_pool_stats_available_bytes
(gauge)
Available storage space in bytes for this pool
Shown as byte
portworx.px_pool_stats_used_bytes
(gauge)
Used storage space in bytes for this pool
Shown as byte
portworx.px_pool_stats_total_bytes
(gauge)
Total storage space in bytes for this pool
Shown as byte
portworx.px_pool_stats_written_bytes_total
(count)
Total bytes written for this pool
Shown as byte
portworx.px_pool_stats_flushed_bytes_total
(count)
Total number of flushed bytes
Shown as byte
portworx.px_pool_stats_num_flushes_total
(count)
Total number of flush(sync) operations
portworx.px_pool_stats_flushms_total
(count)
Total time spent in flush
Shown as millisecond
portworx.px_proc_stats_virt
(gauge)
Virtual memory in bytes
Shown as byte
portworx.px_proc_stats_res
(gauge)
Resident set size memory in bytes
Shown as byte
portworx.px_proc_stats_cputime
(gauge)
Amount of time that this process has been scheduled in user and kernel mode measured in clock ticks
portworx.px_px_cache_status
(gauge)
Cache enabled (0=No,1=Yes)
portworx.px_px_cache_total_blocks
(gauge)
Number of total blocks in the cache
portworx.px_px_cache_used_blocks
(gauge)
Number of used blocks in the cache
portworx.px_px_cache_dirty_blocks
(gauge)
Number of dirty blocks in the cache
portworx.px_px_cache_read_hits
(gauge)
Number of read hits for the cache
portworx.px_px_cache_read_miss
(gauge)
Number of read misses for the cache
portworx.px_px_cache_write_hits
(gauge)
Number of write hits for the cache
portworx.px_px_cache_write_miss
(gauge)
Number of write misses for the cache
portworx.px_px_cache_block_size
(gauge)
Block size for the cache
portworx.px_px_cache_mode
(gauge)
Mode of the cache
portworx.px_px_cache_migrate_promote
(gauge)
Number of blocks promoted to the cache
portworx.px_px_cache_migrate_demote
(gauge)
Number of block demoted from the cache
portworx.px_px_cache_io_mbps
(gauge)
Approximate cache bandwidth from cache internal actions
portworx.px_rebalance_rebalance_job_state
(gauge)
Rebalance job state (0 = pending, 1 = running, 2 = done, 3 = paused, 4 = cancelled)
portworx.px_rebalance_provision_space_rebalanced_bytes_total
(count)
Total provisioned space rebalanced (only counts add (since remove has equal value as add))
Shown as byte
portworx.px_rebalance_used_space_rebalanced_bytes_total
(count)
Total used space rebalanced (only counts add (since remove has equal value as add))
Shown as byte
portworx.px_rebalance_volumes_rebalanced_total
(count)
Total volumes affected by rebalance operation
portworx.px_rebalance_overloaded_pools_total
(count)
Number of overloaded pools
portworx.px_volume_usage_bytes
(gauge)
Used storage space in bytes for this volume
Shown as byte
portworx.px_volume_capacity_bytes
(gauge)
Configured size in bytes for this volume
Shown as byte
portworx.px_volume_halevel
(gauge)
Configured HA level for this volume
portworx.px_volume_currhalevel
(gauge)
Current HA level for this volume
portworx.px_volume_iopriority
(gauge)
Configured I/O priority for this volume
portworx.px_volume_elapsed_time_since_detached_seconds
(gauge)
Seconds elapsed since the volume is detached
Shown as second
portworx.px_volume_elapsed_time_since_attached_seconds
(gauge)
Seconds elapsed since the volume is attached
Shown as second
portworx.px_volume_attached
(gauge)
Attached state for this volume (0=detached,1=attached)
portworx.px_volume_status
(gauge)
Status for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestatus)
portworx.px_volume_state
(gauge)
State for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestate)
portworx.px_volume_attached_state
(gauge)
Attached state for this volume (valid only if volume is attached) (https://libopenstorage.github.io/w/master.generated-api/#attachstate)
portworx.px_volume_fs_health_status
(gauge)
Filesystem health status for this volume (https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus)
portworx.px_volume_replication_status
(gauge)
Replication Status for this volume (0 : up, 1 : not in quorum, 2 : resync state, 3 : degraded, 4 : detached, 5 : restore)
portworx.px_volume_fs_usage_bytes
(gauge)
Used storage space in bytes as reported by the filesystem for this volume
Shown as byte
portworx.px_volume_fs_capacity_bytes
(gauge)
Total storage space in bytes as reported by the filesystem for this volume
Shown as byte
portworx.px_volume_vol_read_bytes
(gauge)
Number of successfully read bytes during this interval for this volume. Deprecated.
Shown as byte
portworx.px_volume_vol_written_bytes
(gauge)
Number of successfully written bytes during this interval for this volume. Deprecated.
Shown as byte
portworx.px_volume_vol_reads
(gauge)
Number of successfully completed read operations during this interval for this volume. Deprecated.
portworx.px_volume_vol_writes
(gauge)
Number of successfully completed write operations during this interval for this volume. Deprecated.
portworx.px_volume_read_bytes
(gauge)
Number of successfully read bytes during this interval for this volume
Shown as byte
portworx.px_volume_written_bytes
(gauge)
Number of successfully written bytes during this interval for this volume
Shown as byte
portworx.px_volume_reads
(gauge)
Number of successfully completed read operations during this interval for this volume
portworx.px_volume_writes
(gauge)
Number of successfully completed write operations during this interval for this volume
portworx.px_volume_read_bytes_total
(count)
Total number of successfully read bytes for this volume
Shown as byte
portworx.px_volume_written_bytes_total
(count)
Total number of successfully written bytes for this volume
Shown as byte
portworx.px_volume_reads_total
(count)
Total number of successfully completed read operations for this volume
portworx.px_volume_writes_total
(count)
Total number of successfully completed write operations for this volume
portworx.px_volume_iops
(gauge)
Number of successful completed I/O operations per second during this interval for this volume
portworx.px_volume_write_iops
(gauge)
Average number of completed write operations per second for this volume
portworx.px_volume_read_iops
(gauge)
Average number of completed read operations per second for this volume
portworx.px_volume_vol_num_sequential_writes
(gauge)
Number of sequential write I/O operations during this interval for this volume
portworx.px_volume_vol_num_sequential_reads
(gauge)
Number of sequential read I/O operations during this interval for this volume
portworx.px_volume_vol_num_random_writes
(gauge)
Number of random write I/O operations during this interval for this volume
portworx.px_volume_vol_num_random_reads
(gauge)
Number of random read I/O operations during this interval for this volume
portworx.px_volume_depth_io
(gauge)
Number of I/O operations currently in progress for this volume
portworx.px_volume_readthroughput
(gauge)
Number of bytes read per second during this interval for this volume
Shown as byte
portworx.px_volume_writethroughput
(gauge)
Number of bytes written per second during this interval for this volume
Shown as byte
portworx.px_volume_vol_bytes_reclaimed
(gauge)
Number of bytes reclaimed by fstrim operation
Shown as byte
portworx.px_volume_vol_bytes_reclaimable
(gauge)
Number of bytes reclaimable on the volume as seen by fstrim operation
Shown as byte
portworx.px_volume_vol_read_latency_seconds
(gauge)
Average time spent per successfully completed read operation in seconds during this interval for this volume. Deprecated.
Shown as second
portworx.px_volume_vol_write_latency_seconds
(gauge)
Average time spent per successfully completed write operation in seconds during this interval for this volume. Deprecated.
Shown as second
portworx.px_volume_read_latency_seconds
(gauge)
Average time spent per successfully completed read operation in seconds for this volume
Shown as second
portworx.px_volume_write_latency_seconds
(gauge)
Average time spent per successfully completed write operation in seconds for this volume
Shown as second
portworx.px_volume_num_long_reads
(gauge)
Number of long reads for this volume
portworx.px_volume_num_long_writes
(gauge)
Number of long writes for this volume
portworx.px_volume_num_long_flushes
(gauge)
Number of long flushes for this volume
portworx.px_volume_num_db_flushes
(gauge)
Number of DB flushes for this volume
portworx.px_volume_replica_read_bytes_total
(count)
Total number of successfully read bytes for this replica volume
Shown as byte
portworx.px_volume_replica_written_bytes_total
(count)
Total number of successfully written bytes for this replica volume
Shown as byte
portworx.px_volume_replica_reads_total
(count)
Total number of successfully completed read operations for this replica volume
portworx.px_volume_replica_writes_total
(count)
Total number of successfully completed write operations for this replica volume
portworx.px_volume_replica_flushes_total
(count)
Total number of successfully completed flush operations for this replica volume
portworx.px_volume_replica_read_ms_total
(count)
Total time spent doing read operations in milliseconds for this replica volume
Shown as millisecond
portworx.px_volume_replica_write_ms_total
(count)
Total time doing write operations in milliseconds for this replica volume
Shown as millisecond
portworx.px_volume_replica_flush_ms_total
(count)
Total time doing write operations in milliseconds for this replica volume
Shown as millisecond
portworx.px_volume_dev_depth_io
(gauge)
Number of I/O operations currently in progress as reported by the kernel pxd device for this volume
portworx.px_volume_dev_writethroughput
(gauge)
Number of successfully written bytes per second as reported by the kernel pxd device for this volume
Shown as byte
portworx.px_volume_dev_readthroughput
(gauge)
Number of successfully read bytes per second as reported by the kernel pxd device for this volume
Shown as byte
portworx.px_volume_dev_read_latency_seconds
(gauge)
Average time spent per successfully completed read in seconds as reported by the kernel pxd device for this volume
Shown as second
portworx.px_volume_dev_write_latency_seconds
(gauge)
Average time spent per successfully completed write in seconds as reported by the kernel pxd device for this volume
Shown as second
portworx.px_volume_dev_read_bytes_total
(count)
Total number of successfully read bytes as reported by the kernel pxd device for this volume
Shown as byte
portworx.px_volume_dev_written_bytes_total
(count)
Number of successfully written bytes as reported by the kernel pxd device for this volume
Shown as byte
portworx.px_volume_dev_reads_total
(count)
Total number of successfully completed read operations as reported by the kernel pxd device for this volume
portworx.px_volume_dev_writes_total
(count)
Total number of successfully completed write operations as reported by the kernel pxd device for this volume
portworx.px_volume_dev_read_seconds_total
(count)
Total time spent reading in seconds for this disk as reported by the kernel pxd device for this volume
Shown as second
portworx.px_volume_dev_write_seconds_total
(count)
Total time spent writing in seconds for this disk as reported by the kernel pxd device for this volume
Shown as second
portworx.px_volume_unique_blocks
(gauge)
Size(in bytes) of unique blocks for this volume
Shown as byte
portworx.px_volume_timestamp_records
(gauge)
Number of timestamp records accumulated
portworx.px_volume_timestamp_records_per_node
(gauge)
Number of timestamp records accumulated for a node
portworx.px_volume_num_discard_ops
(gauge)
Number of discard operations for this volume
portworx.px_volume_discarded_bytes
(gauge)
Number of discarded bytes for this volume
Shown as byte
portworx.autopilot_errors_total
(count)
Total number of autopilot errors
portworx.autopilot_action_failed_count
(gauge)
Total number of declined or failed actions
portworx.autopilot_action_failed_total
(count)
Total number of failed actions per rule object
portworx.autopilot_objects_in_stable_condition_total
(count)
Total number of objects with stable conditions per rule
portworx.autopilot_action_taken_total
(count)
Total number of actions taken per rule object
portworx.autopilot_objects_total
(count)
Total number of objects per rule
portworx.autopilot_objects_with_rules_total
(count)
Total number of objects matching any rules
portworx.autopilot_rules_in_system
(gauge)
Total number of rules in the system
portworx.autopilot_dynamic_throttle_applied_total
(count)
Total number of dynamic throttles applied
portworx.autopilot_dynamic_throttle_failed_total
(count)
Total number of dynamic throttles failed to apply
portworx.autopilot_dynamic_throttle_skipped_total
(count)
Total number of dynamic throttles skipped to apply
portworx.autopilot_datadog_success_total
(count)
Total number of successful Datadog queries
portworx.autopilot_datadog_failures_total
(count)
Total number of failed Datadog queries
portworx.autopilot_datadog_query_validation_failures_total
(count)
Total number of failed Datadog query validations
portworx.autopilot_datadog_empty_response_total
(count)
Total number of empty Datadog responses
portworx.autopilot_prometheus_empty_response_total
(count)
Total number of empty Prometheus responses
portworx.stork_migration_status
(gauge)
Migration status - used to filter and list migrations by status within the cluster.
portworx.stork_migration_stage
(gauge)
Stage of migration (0 = Initial, 1 = PreExecRule, 2 = PostExecRule, 3 = Volumes, 4 = Applications, 5 = Final).
portworx.stork_migration_duration
(gauge)
Duration of migration in seconds. Stage label values: volume (volume migration) or resource (resource migration).
Shown as second
portworx.stork_migration_duration_seconds
(gauge)
Total time taken by a migration in seconds.
Shown as second
portworx.stork_migration_resources
(gauge)
Count of resources collected for migration (state: planned or migrated).
portworx.stork_migration_volumes
(gauge)
Volumes planned for migration (state: planned or migrated).
portworx.stork_migration_created_total
(count)
Increments when a migration is created.
portworx.stork_migration_completed_total
(count)
Increments when a migration reaches a terminal state (Successful, Failed, or PartialSuccess).
portworx.stork_migration_transfer_rate
(gauge)
Data transfer rate for a single migration (bytes/second).
Shown as byte
portworx.stork_migration_transferred_bytes
(gauge)
Total bytes transferred during a single migration.
Shown as byte
portworx.stork_migration_completion_timestamp_seconds
(gauge)
Timestamp when the latest migration completed for a schedule with the status being Successful, PartialSuccess, or Failed.
Shown as second
portworx.stork_migration_schedule_status
(gauge)
Status of migration schedules.
portworx.stork_migration_schedule_last_run_status
(gauge)
Status of the latest run of each migration schedule.
portworx.stork_action_status
(gauge)
Status of Failover or Failback actions.
portworx.stork_action_created_total
(count)
Increments when a Failover/Failback action is created.
portworx.stork_action_completed_total
(count)
Increments when a Failover/Failback action reaches a terminal state.
portworx.stork_action_completion_timestamp_seconds
(gauge)
Timestamp when a Failover/Failback action completes with the status as Successful, Failed, RollbackSuccessful, or RollbackFailed.
Shown as second
portworx.stork_clusterpair_scheduler_status
(gauge)
Status of the cluster pair’s health with respect to Kubernetes (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode).
portworx.stork_clusterpair_storage_status
(gauge)
Status of the cluster pair’s health with respect to storage provisioner (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode).
portworx.stork_all_namespaces
(gauge)
Number of namespaces in the Kubernetes cluster, including Kubernetes system namespaces.
portworx.stork_protected_namespaces
(gauge)
Count of namespaces being migrated through active migration schedules.
portworx.stork_hyperconverged_pods_total
(gauge)
Number of pods scheduled on a node where replicas for all pod volumes exist (hyper-converged by the Stork scheduler).
portworx.stork_non_hyperconverged_pods_total
(gauge)
Number of pods not hyper-converged by the Stork scheduler.
portworx.stork_semi_hyperconverged_pods_total
(gauge)
Number of pods partially hyper-converged (scheduled on nodes with some but not all volume replicas).

Events

The Portworx integration does not include any events.

Service Checks

The Portworx integration does not include any service checks.

Troubleshooting

Agent cannot scrape metrics

If metrics are not appearing in Datadog, check the following:

  • Verify the ComponentK8sConfig CR was applied and the pods have the expected annotations:
    kubectl get pod <portworx-api-pod> -n portworx -o jsonpath='{.metadata.annotations}'
    
  • Confirm the OpenMetrics endpoints are reachable from within the Agent pod:
    kubectl exec <datadog-agent-pod> -n <datadog-namespace> -- curl http://<node-ip>:17001/metrics
    
  • Check Agent logs for scrape errors:
    kubectl logs <datadog-agent-pod> -n <datadog-namespace>
    

Support

Need help? Contact Portworx support.

Further Reading

Additional helpful documentation, links, and articles: