The data has been produced by monitor-usage.sh<\/a>.<\/p>\n Gnuplot 5<\/strong><\/p>\n Gnuplot is the simplest, with a lot available out of the box. But it is also somewhat archaic and little flexible.<\/p>\n Here is the code:<\/p>\n And here is an example output:<\/p>\n Pros: Feature-rich (support bytes axis, two Y axes, reading data).<\/p>\n Cons: Archaic language, less flexible; I couldn\u2019t stop it from drawing X labels over each other when I had data for a longer period.<\/p>\n Incanter (JFreeChart)<\/strong><\/p>\n Incanter is (was?) the go to tool for data analysis in Clojure and it includes the Java library JFreeChart for charting. There has been some development lately (integrating clojure.core.matrix) but otherwise it is quite stagnating. That\u2019s why there have been newer community efforts.<\/p>\n Here is the first part of the code, that prepares the data for charting (also used for Oz):<\/p>\n And here is the code to format the chart (the most difficult part was to display kB \/ MB \/ GB values on the axis in a nice way; I really missed Gnuplot\u2019s out-of-the-box support here): That is quite insane amount of work, huh? And the result:<\/p>\n vega-lite via Oz<\/strong><\/p>\n Vega and Vega-lite visualization grammars:\u00a0Vega is a declarative format for creating, saving, and sharing visualization designs. With Vega, visualizations are described in JSON, and generate interactive views using either HTML5 Canvas or SVG.<\/p>\n<\/blockquote>\n Vega-lite (and Vega<\/a>) is the new kid on the block. It has learned a lot from D3 and R\u2019s ggplot2\u00a0and is intended for data exploration, interactive charts, and making it possible to combine multiple charts on the same page \u2013 while making this as simple as possible. It is based on the Grammar of Graphics so it is presumabely well thought-through.<\/p>\n Oz<\/a> is a thin wrapper around vega\/vega-lite that makes it possible to interact with the browser from Clojure REPL, with Clojure data.<\/p>\n (Note: Oz currently uses VL v2.6 while the latest VL is v3rc8, which fixes some limitations in v2.)<\/p>\n As mentioned above, we use the same usage-chart-preparation.clj<\/a>\u00a0as the pure-Incanter example to prepare the data for charting. Then, to plot them:<\/p>\n It is more work than in Gnuplot but it provides more value and is much more powerful.<\/p>\n Here is the output (generated with VL v3 so the labels have the same color as the lines):<\/p>\n The awesome thing is that the chart and data can be encoded in an URL so that you can open in in the online Vega Editor and play with it<\/a>.<\/p>\n Pros: Good and powerful design, modern, interactive, promising.<\/p>\n Cons: The community is very small so it is harder to get help, it isn\u2019t so mature yet (e.g. it was impossible to add a legend to my multi-layer chart). You have to transform your data into JSON so it likely isn\u2019t suitable for huge amounts of it.<\/p>\n Other options<\/strong><\/p>\n Clojurewerkz\/envision<\/a> is \u201ca small, easy to use Clojure library for data processing, cleanup and visualisation. [..] Main idea of this library is to make exploratory analysis more interactive and visual, although in programmer\u2019s way.\u201d ClojureWerkz is known for its commitment to project quality and maintenance so that is good, on the other hand the last code change has been 2 years ago.<\/p>\n Outside of Clojure, I would expect Python to have some very good charting libraries.<\/p>\n Nothing is optimal\u00a0 but vega-lite is very promising, I will continue to use it \u2013 and I will also still use Incanter to process and prepare the data.<\/p>\nThe tools<\/h2>\n
\r\n#!\/usr\/bin\/env gnuplot --persist -c\r\n# Plot memory and CPU usage over time. Usage:\r\n# usage-plot.gp <input file> [<output .png file>]\r\n# where the input file has the columns `<unix time> <memory, with m\/g suffix> <% cpu>`\r\n# To create the input file, see https:\/\/gist.github.com\/jakubholynet\/931a3441982c833f5f8fcdcf54d05c91\r\n\r\n# Arguments:\r\ninfile=ARG1\r\noutfile=ARG2\r\nset term x11\r\nset title 'Memory, CPU usage from' . infile\r\nset xdata time\r\nset timefmt \"%s\"\r\nset xlabel \"Time [[hh:]mm:ss]\"\r\nset ylabel \"Memory usage\"\r\nset format y '%.1s%cB'\r\n\r\nset y2label 'CPU usage'\r\nset format y2 '%.0s%%'\r\nset y2tics nomirror\r\nset tics out\r\nset autoscale y\r\nset autoscale y2\r\n\r\n# Credit: Christoph @ https:\/\/stackoverflow.com\/a\/52822256\/204205\r\nresolveUnit(s)=(pos=strstrt(\"kmgtp\",s[strlen(s):*]), real(s)*(1024**pos))\r\n\r\nif (exists(\"outfile\") && strlen(outfile) > 0) {\r\n print \"Outputting to the file \", outfile\r\n set term png # 640,480\r\n set output outfile\r\n}\r\n\r\n# Styling\r\nset style line 1 linewidth 2 linecolor 'blue'\r\nset style line 2 linecolor 'light-green'\r\n#set xtics font \", 10\"\r\nset tics font \", 10\"\r\nset xtics rotate 60 # put label every 60s, make vertical so they don't clash in .png if too many\r\n\r\nplot infile u 1:3 with lp axes x1y2 title \"cpu\" linestyle 2, \\\r\n infile using 1:(resolveUnit(stringcolumn(2))) with linespoints title \"memory\" linestyle 1<\/pre>\n![\"plotting](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/11\/usage-gnuplot.png\")
<\/a><\/p>\n \r\n(ns clj-charting.usage-chart-preparation\r\n (:require\r\n [incanter.core :refer :all]\r\n [incanter.stats :as s]\r\n [incanter.io :as io]))\r\n\r\n(defn- resolve-unit-suffix \r\n \"Replace values such as 333k, 800m, 1.2g with the corresponding value in bytes\"\r\n [val-suffixed]\r\n (if-let [[_ val unit] (and\r\n (string? val-suffixed)\r\n (re-find #\"(\\d+)([kmg])\" val-suffixed))]\r\n (let [order (case unit\r\n \"k\" 1\r\n \"m\" 2\r\n \"g\" 3)\r\n scale (apply * (take order (repeat 1024)))]\r\n (* (Integer\/parseInt val) scale))))\r\n\r\n(defn read-usage-data \r\n \"Read usage data in the form `sec.ns memory_with_scale_suffix CPU_percentage` into a dataset with\r\n `ms memory_in_bytes CPU_percentage`\"\r\n [file]\r\n (let [data (io\/read-dataset\r\n file\r\n :delim \\space)]\r\n (-> data\r\n ;; Memory: from 300m or 1g to a number:\r\n (transform-col\r\n :col1\r\n resolve-unit-suffix)\r\n ;; CPU: From <sec>.<nano> to <ms>:\r\n (transform-col\r\n :col0\r\n #(long (* 1000 %))))))\r\n\r\n(defn moving-window-means\r\n \"Given very scattered data, produce a similar sequence of 'moving window mean' where we\r\n replace each point by the mean of it and the preceding\/following `radius` points.\r\n \"\r\n [radius col]\r\n (let [x' (concat (repeat radius nil) col)\r\n parts (drop-last radius (partition (inc (* 2 radius)) 1 x'))\r\n means (map #(-> (remove nil? %) s\/mean long)\r\n parts)]\r\n means))<\/pre>\n
\r\n(ns clj-charting.incanter\r\n (:require\r\n [incanter.core :refer :all]\r\n [incanter.charts :refer :all]\r\n [clj-charting.usage-chart-preparation :refer [read-usage-data moving-window-means]])\r\n (:import\r\n [org.jfree.chart JFreeChart]\r\n [org.jfree.chart.plot XYPlot]\r\n (org.jfree.chart.axis ValueAxis NumberAxis NumberTickUnit TickUnitSource TickUnit)\r\n (java.text NumberFormat DecimalFormat FieldPosition)))\r\n\r\n(defn merge-y-axis\r\n \"Merge the Y axis of two line \/ time series charts. The former chart will have\r\n the left Y axis, and the latter will have the right. Incanter does not support 2 Y\r\n axes out of the box.\r\n Source: https:\/\/matthewdowney.github.io\/clojure-incanter-plot-multiple-y-axis.html\"\r\n [^JFreeChart chart ^JFreeChart chart-to-merge]\r\n (let [^XYPlot plot (.getPlot chart-to-merge)]\r\n (doto ^XYPlot (.getPlot chart)\r\n (.setRangeAxis 1 (.getRangeAxis plot))\r\n (.setDataset 1 (.getDataset plot))\r\n (.mapDatasetToRangeAxis 1 1)\r\n (.setRenderer 1 (.getRenderer plot)))\r\n (-> (.getPlot chart)\r\n (.getLegendItems)\r\n (.addAll (.getLegendItems plot)))\r\n chart))\r\n\r\n(defn byte-scale \r\n \"For the given number [in bytes] return [scale, scale suffix] so that we can divide it \r\n by the scale and display with the corresponding suffix.\r\n Example: 2333 -> [1024 \\\"kB\\\"]\"\r\n [num]\r\n (let [k 1024\r\n m (int (Math\/pow 1024 2))\r\n g (int (Math\/pow 1024 3))]\r\n (condp <= num\r\n g [g \"GB\"]\r\n m [m \"MB\"]\r\n k [k \"kB\"]\r\n [1 \"\"])))\r\n\r\n(defn format-bytes \r\n \"For the given number [in bytes] return [the number scaled down, the scale suffix such as \\\"kB\\\"].\r\n Example: 2333 -> [2.278 \\\"kB\\\"]\"\r\n [num]\r\n (let [[scale unit] (byte-scale num)]\r\n [(\/ num scale) unit]))\r\n\r\n;; Instance of NumberFormat that displays a byte number scaled down and with the scale suffix\r\n;; Example: 2333 -> \\\"2.3kB\\\"]\"\r\n(def byteFmt (let [dec-fmt (java.text.DecimalFormat. \"#.#\")]\r\n (proxy [java.text.NumberFormat] []\r\n (format [^double number, ^StringBuffer toAppendTo, ^FieldPosition pos]\r\n (let [[n u] (format-bytes number)]\r\n (.append\r\n (.format dec-fmt n toAppendTo pos)\r\n u))))))\r\n\r\n(defn nearest-byte-tick\r\n \"For the given byte number, find out what tick to show on the axis; \r\n e.g. we would rather see a tick such as '800MB' than '783.5MB' on it.\"\r\n ([^double size tick-fn]\r\n (let [[scale] (byte-scale size)]\r\n (NumberTickUnit.\r\n (* scale\r\n ;; FIXME if size = 1000 upgrade to 1024\r\n (.getSize\r\n (tick-fn\r\n (NumberTickUnit. (\/ size scale)))))\r\n byteFmt))))\r\n\r\n(def byte-tick-source\r\n \"TickUnitSource suitable for byte values spanning multiple of kB - MB - GB\"\r\n ;; TODO Instead of reusing IntegerTickUnits, reimplement it to support powers of 2\r\n (let [int-tick-units (NumberAxis\/createIntegerTickUnits)]\r\n (reify\r\n TickUnitSource\r\n (^TickUnit getLargerTickUnit [_ ^TickUnit unit]\r\n (nearest-byte-tick\r\n (.getSize unit)\r\n #(.getLargerTickUnit int-tick-units %)))\r\n (^TickUnit getCeilingTickUnit [me ^TickUnit unit]\r\n (.getCeilingTickUnit me (.getSize unit)))\r\n (^TickUnit getCeilingTickUnit [_ ^double size]\r\n (nearest-byte-tick\r\n size\r\n #(.getCeilingTickUnit int-tick-units %))))))\r\n\r\n(defn set-bytes-tick-unit [^JFreeChart chart]\r\n (let [^XYPlot plot (.getPlot chart)\r\n ^NumberAxis axis (.getRangeAxis plot)]\r\n (.setStandardTickUnits axis byte-tick-source)\r\n chart))\r\n\r\n\r\n(defn plot-usage [file]\r\n (let [data (read-usage-data file)\r\n time (sel data :cols 0)\r\n mem (sel data :cols 1)\r\n cpu (sel data :cols 2)]\r\n (->\r\n (time-series-plot time cpu :title file :y-label \"cpu [%]\" :legend true)\r\n (add-lines time (moving-window-means 60 cpu) :series-label \"cpu (mean)\")\r\n (merge-y-axis\r\n (set-bytes-tick-unit\r\n (time-series-plot time mem :series-label \"Memory\")))\r\n (view))))\r\n\r\n(plot-usage \"siege-c10-all-urls-async-node11.dat\")<\/pre>\n
![\"plotting](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/11\/cpu-usage-with-async-hooks-node11.png\")
<\/a><\/p>\n\n
\r\n(ns clj-charting.oz\r\n (:require\r\n [oz.core :as oz]\r\n [incanter.core :refer :all]\r\n [clj-charting.usage-chart-preparation :refer [read-usage-data moving-window-means]]))\r\n\r\n(defn dataset->map-list \r\n \"Incanter dataset into a list of maps like \r\n {\\\"0\\\" 1541065398391, \\\"1\\\" 446693376, \\\"2\\\" 99.9, \\\"cpu_mean\\\" 89}\"\r\n [ds]\r\n (let [rows (to-list ds)\r\n means (moving-window-means 60 (sel ds :cols 2))]\r\n (map\r\n #(assoc\r\n (zipmap (map str (range)) %1)\r\n \"cpu_mean\" %2)\r\n rows\r\n means)))\r\n\r\n;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\r\n\r\n(def gb4 (* 3 1024 1024 1024))\r\n\r\n;; TODO Display legend - IMPOSSIBLE :-( until Datum\r\n(def line-plot\r\n (let [data (dataset->map-list (read-usage-data \"siege-c10-all-urls-async-node11.dat\"))\r\n x-enc {:field \"0\"\r\n :type \"temporal\"\r\n :timeUnit \"hoursminutesseconds\" ; :aggregate \"mean\" l <- this kills points with same value\r\n :axis {:title \"Time\"}\r\n :scale {:zero false}}]\r\n {:width 700\r\n :data {:values data}\r\n\r\n ;;; ?? requires VL v3 until then we have to compute cpu_mean using Incanter\r\n ;:transform [{:window [{:op \"mean\"\r\n ; :field \"1\"\r\n ; :as \"cpu_mean\"}]\r\n ; :frame [-10, 10]}]\r\n ; TODO VLv3: use this ?? instead of repeating the X on each plot\r\n ;:encoding {:x x-enc}\r\n :layer [{:mark {:type \"line\"\r\n :clip true\r\n :color \"red\"}\r\n :encoding {:x x-enc\r\n :y {:field \"1\"\r\n :type \"quantitative\"\r\n :axis {:format \".1s\" :title \"Memory\" :labelColor \"red\" #_\"required VL 3\"}\r\n :scale {:domain [0 gb4]}}}}\r\n {:layer [\r\n {:mark {:type \"point\"\r\n :clip true}\r\n :encoding {:x x-enc\r\n :y {:field \"2\"\r\n :type \"quantitative\"\r\n :axis {:title \"CPU [%]\" :labelColor \"blue\"}}}}\r\n {:mark {:type \"line\"\r\n :clip true\r\n :color \"blue\"}\r\n :encoding {:x x-enc\r\n :y {:field \"cpu_mean\"\r\n :type \"quantitative\"\r\n :title nil\r\n :axis nil}}}]}]\r\n :resolve {:scale {:y \"independent\"}}}))\r\n\r\n(oz\/start-plot-server!)\r\n(oz\/v! line-plot)<\/pre>\n![\"plotting](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/11\/usage-vega-lite-v2.png\")
<\/a><\/p>\nConclusion<\/h2>\n