--web.enable-otlp-receiver<\/code><\/li>\n- Native histograms<\/strong> – A new histogram representation that reduces storage and query cost significantly compared to classic histograms<\/li>\n
- Remote Write 2.0<\/strong> – Improved protocol for sending metrics to remote storage backends with better efficiency<\/li>\n
- UTF-8 metric names<\/strong> – Metric and label names can now contain UTF-8 characters<\/li>\n
- Removed console templates<\/strong> – The
consoles\/<\/code> and console_libraries\/<\/code> directories are no longer included in the tarball<\/li>\n<\/ul>\n\n\n\nFor the full list of changes, see the official Prometheus 3.0 announcement<\/a>.<\/p>\n\n\n\nPrerequisites<\/h2>\n\n\n\n
Before starting, make sure you have:<\/p>\n\n\n\n
\n- Ubuntu 24.04 or Debian 13 server with root or sudo access<\/li>\n
- At least 2 GB RAM and 20 GB free disk space<\/li>\n
- Firewall access to ports 9090 (Prometheus) and 9100 (node_exporter)<\/li>\n
- A working Grafana instance – see Install Grafana on Ubuntu \/ Debian<\/a> if you need one<\/li>\n<\/ul>\n\n\n\n
Step 1: Install Prometheus 3<\/h2>\n\n\n\n
Prometheus ships as a single static binary. We’ll download the latest release directly from GitHub and set it up with a dedicated system user.<\/p>\n\n\n\n
Create the Prometheus user and directories<\/h3>\n\n\n\n
First, create a dedicated user for Prometheus with no login shell and no home directory. This follows the principle of least privilege for running services.<\/p>\n\n\n\n
sudo useradd --no-create-home --shell \/bin\/false prometheus<\/code><\/pre>\n\n\n\nCreate the configuration and data directories:<\/p>\n\n\n\n
sudo mkdir -p \/etc\/prometheus \/var\/lib\/prometheus\nsudo chown prometheus:prometheus \/var\/lib\/prometheus<\/code><\/pre>\n\n\n\nDownload and install Prometheus<\/h3>\n\n\n\n
Fetch the latest version number dynamically from GitHub and download the appropriate binary:<\/p>\n\n\n\n
VER=$(curl -sI https:\/\/github.com\/prometheus\/prometheus\/releases\/latest | grep -i ^location | grep -o v[0-9.]* | sed s\/^v\/\/)\necho \"Installing Prometheus $VER\"\nwget https:\/\/github.com\/prometheus\/prometheus\/releases\/download\/v${VER}\/prometheus-${VER}.linux-amd64.tar.gz<\/code><\/pre>\n\n\n\nAt the time of writing, this pulls Prometheus 3.10.0<\/strong>. Extract the archive and move the binaries into place:<\/p>\n\n\n\ntar xvf prometheus-${VER}.linux-amd64.tar.gz\nsudo cp prometheus-${VER}.linux-amd64\/{prometheus,promtool} \/usr\/local\/bin\/\nsudo chown prometheus:prometheus \/usr\/local\/bin\/{prometheus,promtool}<\/code><\/pre>\n\n\n\nVerify the installation by checking the version:<\/p>\n\n\n\n
prometheus --version<\/code><\/pre>\n\n\n\nYou should see the version and build info confirmed:<\/p>\n\n\n\n
prometheus, version 3.10.0 (branch: HEAD, revision: 4e546fca)\n build user: root@bba5c76a3954\n build date: 20260315-10:42:25\n go version: go1.24.1\n platform: linux\/amd64\n tags: netgo,builtinassets,stringlabels<\/code><\/pre>\n\n\n\nNote:<\/strong> If you’re coming from Prometheus 2.x, notice that the tarball no longer includes consoles\/<\/code> and console_libraries\/<\/code> directories. The new UI replaces the old console templates entirely, so the --web.console.templates<\/code> and --web.console.libraries<\/code> flags are no longer needed.<\/p>\n\n\n\nStep 2: Configure Prometheus<\/h2>\n\n\n\n
Create the main Prometheus configuration file. This config sets up scrape targets for Prometheus itself, node_exporter, Blackbox exporter, and Alertmanager.<\/p>\n\n\n\n
sudo vi \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n\n\n\nAdd the following configuration:<\/p>\n\n\n\n
global:\n scrape_interval: 15s\n evaluation_interval: 15s\n scrape_timeout: 10s\n\nalerting:\n alertmanagers:\n - static_configs:\n - targets:\n - localhost:9093\n\nrule_files:\n - \"\/etc\/prometheus\/rules\/*.yml\"\n\nscrape_configs:\n - job_name: \"prometheus\"\n static_configs:\n - targets: [\"localhost:9090\"]\n\n - job_name: \"node_exporter\"\n static_configs:\n - targets: [\"localhost:9100\"]\n labels:\n instance_name: \"prometheus-server\"\n\n - job_name: \"blackbox_http\"\n metrics_path: \/probe\n params:\n module: [http_2xx]\n static_configs:\n - targets:\n - https:\/\/computingforgeeks.com\n - https:\/\/google.com\n - https:\/\/github.com\n relabel_configs:\n - source_labels: [__address__]\n target_label: __param_target\n - source_labels: [__param_target]\n target_label: instance\n - target_label: __address__\n replacement: localhost:9115\n\n - job_name: \"alertmanager\"\n static_configs:\n - targets: [\"localhost:9093\"]<\/code><\/pre>\n\n\n\nSet the correct ownership on the config file:<\/p>\n\n\n\n
sudo chown prometheus:prometheus \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n\n\n\nStep 3: Create Alert Rules<\/h2>\n\n\n\n
Production monitoring is useless without proper alerting. Create a rules directory and add alert rules for the most critical conditions.<\/p>\n\n\n\n
sudo mkdir -p \/etc\/prometheus\/rules\nsudo vi \/etc\/prometheus\/rules\/node-alerts.yml<\/code><\/pre>\n\n\n\nAdd these production-tested alert rules:<\/p>\n\n\n\n
groups:\n - name: node_alerts\n rules:\n - alert: InstanceDown\n expr: up == 0\n for: 2m\n labels:\n severity: critical\n annotations:\n summary: \"Instance {{ $labels.instance }} is down\"\n description: \"{{ $labels.job }} target {{ $labels.instance }} has been unreachable for more than 2 minutes.\"\n\n - alert: HighCPUUsage\n expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100) > 85\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"High CPU usage on {{ $labels.instance }}\"\n description: \"CPU usage has been above 85% for 5 minutes. Current value: {{ $value | printf \\\"%.1f\\\" }}%\"\n\n - alert: HighMemoryUsage\n expr: (1 - (node_memory_MemAvailable_bytes \/ node_memory_MemTotal_bytes)) * 100 > 90\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"High memory usage on {{ $labels.instance }}\"\n description: \"Memory usage has been above 90% for 5 minutes. Current value: {{ $value | printf \\\"%.1f\\\" }}%\"\n\n - alert: DiskSpaceLow\n expr: (1 - (node_filesystem_avail_bytes{mountpoint=\"\/\"} \/ node_filesystem_size_bytes{mountpoint=\"\/\"})) * 100 > 85\n for: 10m\n labels:\n severity: warning\n annotations:\n summary: \"Disk space low on {{ $labels.instance }}\"\n description: \"Root filesystem usage is above 85%. Current value: {{ $value | printf \\\"%.1f\\\" }}%\"<\/code><\/pre>\n\n\n\nValidate the rules file using promtool:<\/p>\n\n\n\n
promtool check rules \/etc\/prometheus\/rules\/node-alerts.yml<\/code><\/pre>\n\n\n\nA clean validation looks like this:<\/p>\n\n\n\n
Checking \/etc\/prometheus\/rules\/node-alerts.yml\n SUCCESS: 4 rules found<\/code><\/pre>\n\n\n\nSet ownership on the rules directory:<\/p>\n\n\n\n
sudo chown -R prometheus:prometheus \/etc\/prometheus\/rules<\/code><\/pre>\n\n\n\nStep 4: Create the Systemd Service<\/h2>\n\n\n\n
Create a systemd unit file that runs Prometheus with production-ready flags, including OTLP support and a 30-day data retention period.<\/p>\n\n\n\n
sudo vi \/etc\/systemd\/system\/prometheus.service<\/code><\/pre>\n\n\n\nAdd the following service definition:<\/p>\n\n\n\n
[Unit]\nDescription=Prometheus 3 Monitoring System\nDocumentation=https:\/\/prometheus.io\/docs\/introduction\/overview\/\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=simple\nUser=prometheus\nGroup=prometheus\nExecReload=\/bin\/kill -HUP $MAINPID\nExecStart=\/usr\/local\/bin\/prometheus \\\n --config.file=\/etc\/prometheus\/prometheus.yml \\\n --storage.tsdb.path=\/var\/lib\/prometheus \\\n --storage.tsdb.retention.time=30d \\\n --web.listen-address=0.0.0.0:9090 \\\n --web.enable-lifecycle \\\n --web.enable-otlp-receiver\nRestart=always\nRestartSec=5\nSyslogIdentifier=prometheus\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\nThe --web.enable-otlp-receiver<\/code> flag is new in Prometheus 3 and lets you ingest metrics from OpenTelemetry collectors directly into Prometheus. The --web.enable-lifecycle<\/code> flag allows reloading the config via HTTP POST to \/-\/reload<\/code>.<\/p>\n\n\n\nStart and enable the service:<\/p>\n\n\n\n
sudo systemctl daemon-reload\nsudo systemctl enable --now prometheus<\/code><\/pre>\n\n\n\nVerify Prometheus is running:<\/p>\n\n\n\n
sudo systemctl status prometheus<\/code><\/pre>\n\n\n\nThe service should show active (running) with the TSDB successfully opened:<\/p>\n\n\n\n
\u25cf prometheus.service - Prometheus 3 Monitoring System\n Loaded: loaded (\/etc\/systemd\/system\/prometheus.service; enabled; preset: enabled)\n Active: active (running) since Mon 2026-03-24 10:15:23 UTC; 5s ago\n Docs: https:\/\/prometheus.io\/docs\/introduction\/overview\/\n Main PID: 2341 (prometheus)\n Tasks: 8 (limit: 4557)\n Memory: 42.3M\n CPU: 1.245s\n CGroup: \/system.slice\/prometheus.service\n \u2514\u25002341 \/usr\/local\/bin\/prometheus --config.file=\/etc\/prometheus\/prometheus.yml ...<\/code><\/pre>\n\n\n\nStep 5: Install Node Exporter<\/h2>\n\n\n\n
Node exporter collects hardware and OS-level metrics from the host – CPU, memory, disk, network, filesystem, and more. It’s the most common exporter and should run on every server you monitor.<\/p>\n\n\n\n
Create a dedicated user for node_exporter:<\/p>\n\n\n\n
sudo useradd --no-create-home --shell \/bin\/false node_exporter<\/code><\/pre>\n\n\n\nDownload and install the latest node_exporter release:<\/p>\n\n\n\n
VER=$(curl -sI https:\/\/github.com\/prometheus\/node_exporter\/releases\/latest | grep -i ^location | grep -o v[0-9.]* | sed s\/^v\/\/)\necho \"Installing node_exporter $VER\"\nwget https:\/\/github.com\/prometheus\/node_exporter\/releases\/download\/v${VER}\/node_exporter-${VER}.linux-amd64.tar.gz\ntar xvf node_exporter-${VER}.linux-amd64.tar.gz\nsudo cp node_exporter-${VER}.linux-amd64\/node_exporter \/usr\/local\/bin\/\nsudo chown node_exporter:node_exporter \/usr\/local\/bin\/node_exporter<\/code><\/pre>\n\n\n\nAt the time of writing, this installs node_exporter 1.10.2<\/strong>. Create the systemd service file:<\/p>\n\n\n\nsudo vi \/etc\/systemd\/system\/node_exporter.service<\/code><\/pre>\n\n\n\nAdd the following service definition with extra collectors enabled:<\/p>\n\n\n\n
[Unit]\nDescription=Prometheus Node Exporter\nDocumentation=https:\/\/github.com\/prometheus\/node_exporter\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=simple\nUser=node_exporter\nGroup=node_exporter\nExecStart=\/usr\/local\/bin\/node_exporter \\\n --collector.systemd \\\n --collector.processes\nRestart=always\nRestartSec=5\nSyslogIdentifier=node_exporter\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\nThe --collector.systemd<\/code> flag exposes systemd service states as metrics, which is very useful for alerting on failed services. The --collector.processes<\/code> flag provides process-level metrics including counts by state.<\/p>\n\n\n\nStart and enable node_exporter:<\/p>\n\n\n\n
sudo systemctl daemon-reload\nsudo systemctl enable --now node_exporter<\/code><\/pre>\n\n\n\nConfirm it’s running and listening on port 9100:<\/p>\n\n\n\n
sudo systemctl status node_exporter<\/code><\/pre>\n\n\n\nThe service should show active (running):<\/p>\n\n\n\n
\u25cf node_exporter.service - Prometheus Node Exporter\n Loaded: loaded (\/etc\/systemd\/system\/node_exporter.service; enabled; preset: enabled)\n Active: active (running) since Mon 2026-03-24 10:18:45 UTC; 3s ago\n Docs: https:\/\/github.com\/prometheus\/node_exporter\n Main PID: 2567 (node_exporter)\n Tasks: 5 (limit: 4557)\n Memory: 12.1M\n CPU: 0.089s\n CGroup: \/system.slice\/node_exporter.service\n \u2514\u25002567 \/usr\/local\/bin\/node_exporter --collector.systemd --collector.processes<\/code><\/pre>\n\n\n\nTest that metrics are being exposed by curling the metrics endpoint:<\/p>\n\n\n\n
curl -s http:\/\/localhost:9100\/metrics | head -5<\/code><\/pre>\n\n\n\nYou should see metric lines starting with #<\/code> comments and metric values:<\/p>\n\n\n\n# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=\"0\"} 2.3457e-05\ngo_gc_duration_seconds{quantile=\"0.25\"} 3.1245e-05\ngo_gc_duration_seconds{quantile=\"0.5\"} 4.5678e-05<\/code><\/pre>\n\n\n\nStep 6: Configure Firewall Rules<\/h2>\n\n\n\n
Open the necessary ports in UFW for Prometheus and node_exporter:<\/p>\n\n\n\n
sudo ufw allow 9090\/tcp comment \"Prometheus\"\nsudo ufw allow 9100\/tcp comment \"Node Exporter\"\nsudo ufw reload<\/code><\/pre>\n\n\n\nVerify the rules are active:<\/p>\n\n\n\n
sudo ufw status verbose<\/code><\/pre>\n\n\n\nThe output should list both ports as ALLOW:<\/p>\n\n\n\n
Status: active\n\nTo Action From\n-- ------ ----\n22\/tcp ALLOW Anywhere\n9090\/tcp ALLOW Anywhere # Prometheus\n9100\/tcp ALLOW Anywhere # Node Exporter<\/code><\/pre>\n\n\n\nIn production, restrict port 9100 to only your Prometheus server IP instead of allowing it from anywhere. Node exporter exposes detailed system information that you don’t want publicly accessible.<\/p>\n\n\n\n
Step 7: Access the Prometheus 3 Web UI<\/h2>\n\n\n\n
Open your browser and navigate to http:\/\/your-server-ip:9090<\/code>. The Prometheus 3 UI has been completely redesigned with a modern interface.<\/p>\n\n\n\nNavigate to Status > Targets<\/strong> to verify all configured targets are being scraped successfully. All targets should show a green “UP” state:<\/p>\n\n\n\n![\"Prometheus](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/prometheus3-targets.png\" "\\"\\"")
Prometheus 3 targets page – all endpoints reporting UP<\/figcaption><\/figure>\n\n\n\nTry a basic query in the new query interface. Enter up<\/code> in the expression editor and click Execute to see the status of all targets:<\/p>\n\n\n\n![\"Prometheus](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/prometheus3-query-ui.png\" "\\"\\"")
The redesigned query UI in Prometheus 3 with autocomplete support<\/figcaption><\/figure>\n\n\n\nCheck the Alerts<\/strong> tab to confirm your alert rules are loaded. They should show as inactive (green) when conditions are healthy:<\/p>\n\n\n\n![\"Prometheus](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/prometheus3-alerts.png\" "\\"\\"")
Alert rules loaded and showing inactive (healthy) state<\/figcaption><\/figure>\n\n\n\nStep 8: Connect Grafana to Prometheus<\/h2>\n\n\n\n
With Prometheus collecting metrics, the next step is visualizing them in Grafana. If you don’t have Grafana installed yet, follow our Grafana installation guide for Ubuntu\/Debian<\/a> first.<\/p>\n\n\n\nAdd Prometheus as a data source<\/h3>\n\n\n\n
In Grafana, go to Connections > Data Sources > Add data source<\/strong> and select Prometheus. Set the connection URL to http:\/\/localhost:9090<\/code> (or your Prometheus server IP if Grafana runs on a different host). Click Save & Test<\/strong> to verify the connection.<\/p>\n\n\n\nImport the Node Exporter Full dashboard<\/h3>\n\n\n\n
The community-maintained “Node Exporter Full” dashboard (ID: 1860<\/strong>) is the most popular dashboard for node_exporter metrics. It provides comprehensive views of CPU, memory, disk, network, and filesystem usage.<\/p>\n\n\n\nGo to Dashboards > New > Import<\/strong>, enter dashboard ID 1860<\/code>, select your Prometheus data source, and click Import. The dashboard should immediately populate with live data from your server.<\/p>\n\n\n\n![\"Grafana](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/grafana-node-exporter-dashboard.png\" "\\"\\"")
Node Exporter Full dashboard in Grafana with live metrics from the Prometheus server<\/figcaption><\/figure>\n\n\n\nScroll down to see detailed CPU and memory usage panels with historical data:<\/p>\n\n\n\n![\"Grafana](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/grafana-node-exporter-cpu-mem.png\" "\\"\\"")
Detailed CPU and memory panels from the Node Exporter dashboard<\/figcaption><\/figure>\n\n\n\nThe TSDB Status page under Status > TSDB Status<\/strong> shows useful storage statistics including the number of time series, label pair cardinality, and memory usage:<\/p>\n\n\n\n![\"Prometheus](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/prometheus3-tsdb-status.png\" "\\"\\"")
TSDB status page showing storage statistics and series cardinality<\/figcaption><\/figure>\n\n\n\nUseful PromQL Queries to Get Started<\/h2>\n\n\n\n
With Prometheus collecting node_exporter metrics, here are some essential PromQL queries you can run in the query UI or use in Grafana dashboards:<\/p>\n\n\n\n
Current CPU usage percentage (averaged across all cores):<\/strong><\/p>\n\n\n\n100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)<\/code><\/pre>\n\n\n\nMemory usage percentage:<\/strong><\/p>\n\n\n\n(1 - (node_memory_MemAvailable_bytes \/ node_memory_MemTotal_bytes)) * 100<\/code><\/pre>\n\n\n\nRoot filesystem usage percentage:<\/strong><\/p>\n\n\n\n(1 - (node_filesystem_avail_bytes{mountpoint=\"\/\"} \/ node_filesystem_size_bytes{mountpoint=\"\/\"})) * 100<\/code><\/pre>\n\n\n\nNetwork traffic rate (bytes per second on all interfaces):<\/strong><\/p>\n\n\n\nrate(node_network_receive_bytes_total{device!=\"lo\"}[5m])<\/code><\/pre>\n\n\n\nSystem uptime in days:<\/strong><\/p>\n\n\n\n(time() - node_boot_time_seconds) \/ 86400<\/code><\/pre>\n\n\n\nThese queries form the foundation of most monitoring dashboards. The rate()<\/code> function is essential for counter metrics – it calculates the per-second rate of increase, which is what you actually want to see for counters like CPU seconds and network bytes.<\/p>\n\n\n\nHow to Reload Configuration Without Restarting<\/h2>\n\n\n\n
Prometheus supports hot reloading of configuration. After editing prometheus.yml<\/code> or alert rules, you can reload without downtime using either method:<\/p>\n\n\n\nsudo systemctl reload prometheus<\/code><\/pre>\n\n\n\nOr send an HTTP POST to the lifecycle endpoint (requires --web.enable-lifecycle<\/code>):<\/p>\n\n\n\ncurl -X POST http:\/\/localhost:9090\/-\/reload<\/code><\/pre>\n\n\n\nAlways validate your config before reloading to avoid breaking a running instance:<\/p>\n\n\n\n
promtool check config \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n\n\n\nA successful validation returns:<\/p>\n\n\n\n
Checking \/etc\/prometheus\/prometheus.yml\n SUCCESS: \/etc\/prometheus\/prometheus.yml is valid prometheus config file<\/code><\/pre>\n\n\n\nHow to Add More Monitoring Targets<\/h2>\n\n\n\n
To monitor additional servers, install node_exporter on each target host and add them to the node_exporter<\/code> job in prometheus.yml<\/code>. Each target is just an IP:port entry in the static_configs targets list:<\/p>\n\n\n\n - job_name: \"node_exporter\"\n static_configs:\n - targets: [\"localhost:9100\"]\n labels:\n instance_name: \"prometheus-server\"\n - targets: [\"10.0.1.20:9100\"]\n labels:\n instance_name: \"web-01\"\n - targets: [\"10.0.1.21:9100\"]\n labels:\n instance_name: \"web-02\"\n - targets: [\"10.0.1.30:9100\"]\n labels:\n instance_name: \"db-01\"<\/code><\/pre>\n\n\n\nAfter adding new targets, validate and reload:<\/p>\n\n\n\n
promtool check config \/etc\/prometheus\/prometheus.yml\nsudo systemctl reload prometheus<\/code><\/pre>\n\n\n\nFor large environments with many targets, consider using file-based service discovery instead of static configs. Create a JSON file with your targets and reference it in the scrape job – Prometheus watches the file for changes and picks up new targets automatically without reloading.<\/p>\n\n\n\n
How to Secure Prometheus in Production<\/h2>\n\n\n\n
Prometheus 3 supports basic authentication and TLS natively. For production deployments, you should enable both to prevent unauthorized access to your monitoring data.<\/p>\n\n\n\n
Create a web configuration file for Prometheus:<\/p>\n\n\n\n
sudo vi \/etc\/prometheus\/web.yml<\/code><\/pre>\n\n\n\nAdd basic authentication (generate the bcrypt hash with htpasswd -nBC 10 admin<\/code>):<\/p>\n\n\n\nbasic_auth_users:\n admin: '$2y$10$example_bcrypt_hash_here'<\/code><\/pre>\n\n\n\nThen add --web.config.file=\/etc\/prometheus\/web.yml<\/code> to your systemd service ExecStart. After restarting, the Prometheus UI and API will require authentication. Update your Grafana data source configuration to include the username and password.<\/p>\n\n\n\nFor TLS, add the certificate paths to the same web.yml file. This is especially important if Prometheus is accessible over a network rather than just localhost.<\/p>\n\n\n\n
Using Grafana Alloy as an Alternative Collector<\/h2>\n\n\n\n
If you’re looking for a more flexible metrics collection agent that supports OpenTelemetry natively, consider Grafana Alloy<\/a>. Alloy can replace node_exporter and push metrics to Prometheus using remote write, which is useful in environments where you can’t open inbound ports on target hosts.<\/p>\n\n\n\nTroubleshooting Common Issues<\/h2>\n\n\n\nPrometheus fails to start with “permission denied”<\/h3>\n\n\n\n
If Prometheus can’t write to its data directory, check the ownership:<\/p>\n\n\n\n
ls -la \/var\/lib\/prometheus<\/code><\/pre>\n\n\n\nThe directory must be owned by the prometheus user. Fix it with:<\/p>\n\n\n\n
sudo chown -R prometheus:prometheus \/var\/lib\/prometheus<\/code><\/pre>\n\n\n\nTargets show as DOWN in the UI<\/h3>\n\n\n\n
When a target shows DOWN, click on the error message in the Targets page. Common causes:<\/p>\n\n\n\n
\n- “connection refused”<\/strong> – The exporter service is not running. Check with
systemctl status node_exporter<\/code><\/li>\n- “context deadline exceeded”<\/strong> – The target is too slow to respond. Increase
scrape_timeout<\/code> in the job config<\/li>\n- “no route to host”<\/strong> – Firewall is blocking the port. Verify with
sudo ufw status<\/code><\/li>\n<\/ul>\n\n\n\nOld console flags cause startup errors<\/h3>\n\n\n\n
If you’re migrating from Prometheus 2.x and your systemd service includes --web.console.templates<\/code> or --web.console.libraries<\/code> flags, Prometheus 3 will fail to start. Remove these flags from your service file – they are no longer supported since the console templates were removed from the distribution.<\/p>\n\n\n\nHigh memory usage on large installations<\/h3>\n\n\n\n
Prometheus stores recent data in memory before flushing to disk. If you have many targets or high-cardinality metrics, memory usage can grow quickly. Reduce cardinality by dropping unnecessary labels in your scrape configs, or reduce scrape_interval<\/code> for less critical targets. You can monitor Prometheus’s own memory usage with:<\/p>\n\n\n\nprocess_resident_memory_bytes{job=\"prometheus\"}<\/code><\/pre>\n\n\n\nHow to Check Prometheus Storage Usage<\/h2>\n\n\n\n
Prometheus stores time-series data on disk in the TSDB. Monitoring the storage consumption helps you plan capacity and adjust retention settings. Check the current disk usage:<\/p>\n\n\n\n
du -sh \/var\/lib\/prometheus<\/code><\/pre>\n\n\n\nYou can also monitor storage via PromQL. This query shows the total number of time series currently stored:<\/p>\n\n\n\n
prometheus_tsdb_head_series<\/code><\/pre>\n\n\n\nAnd this shows the rate of samples ingested per second:<\/p>\n\n\n\n
rate(prometheus_tsdb_head_samples_appended_total[5m])<\/code><\/pre>\n\n\n\nA rough estimate for storage: each time series consumes about 1-2 bytes per sample. With a 15-second scrape interval and 30-day retention, expect roughly 3-6 KB per time series per day. If you have 10,000 active series, that’s 30-60 MB per day or about 1-2 GB per month. Adjust --storage.tsdb.retention.time<\/code> based on your available disk space and how far back you need to query.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\n
You now have a production-ready Prometheus 3 installation with node_exporter collecting system metrics, alert rules watching for common issues, and Grafana dashboards providing visualization. The OTLP receiver is enabled for future OpenTelemetry integration. From here, add more exporters for your specific services – database exporters, web server exporters, or custom application metrics – and expand your alert rules to match your SLAs.<\/p>\n","protected":false},"excerpt":{"rendered":"
Prometheus 3 brings a completely redesigned UI, native OpenTelemetry support, and significant performance improvements over the 2.x series. This guide walks through a production-ready installation of Prometheus 3 with node_exporter on Ubuntu 24.04 and Debian 13, including alert rules, Grafana integration, and real-world troubleshooting tips. What’s New in Prometheus 3? Prometheus 3 is a major … Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":164042,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26,299,50,165,81],"tags":[],"class_list":["post-164041","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-debian","category-how-to","category-linux-tutorials","category-monitoring","category-ubuntu"],"_links":{"self":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/164041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/comments?post=164041"}],"version-history":[{"count":0,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/164041\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media\/164042"}],"wp:attachment":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media?parent=164041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/categories?post=164041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/tags?post=164041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}