{"id":106881,"date":"2020-09-25T07:00:00","date_gmt":"2020-09-25T04:00:00","guid":{"rendered":"https:\/\/www.javacodegeeks.com\/?p=106881"},"modified":"2020-09-23T16:53:52","modified_gmt":"2020-09-23T13:53:52","slug":"command-line-json-processing-with-jq","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2020\/09\/command-line-json-processing-with-jq.html","title":{"rendered":"Command-line JSON processing with jq"},"content":{"rendered":"

In this post we will learn how to parse, pretty-print and process JSON from the command-line with jq<\/a>. At the end we will even use jq to do a simple JSON to CSV conversion. jq describes itself as a lightweight and flexible command-line JSON processor<\/em>. You can think of unix shell tools like sed, grep and awk but for JSON.<\/p>\n

jq works on various platforms. Prebuild binaries are available for Linux, Windows and Mac OS. See the jq download site<\/a> for instructions.<\/p>\n

For many of the following examples we will use a file named artist.json<\/em> with the following JSON content:<\/p>\n

\n
\n\n\n\n
\n
01<\/div>\n
02<\/div>\n
03<\/div>\n
04<\/div>\n
05<\/div>\n
06<\/div>\n
07<\/div>\n
08<\/div>\n
09<\/div>\n
10<\/div>\n
11<\/div>\n<\/td>\n
\n
\n
{<\/code><\/div>\n
    <\/code>\"name\"<\/code>: <\/code>\"Leonardo da Vinci\"<\/code>,<\/code><\/div>\n
    <\/code>\"artworks\"<\/code>: [{<\/code><\/div>\n
            <\/code>\"name\"<\/code>: <\/code>\"Mona Lisa\"<\/code>,<\/code><\/div>\n
            <\/code>\"type\"<\/code>: <\/code>\"Painting\"<\/code><\/div>\n
        <\/code>}, {<\/code><\/div>\n
            <\/code>\"name\"<\/code>: <\/code>\"The Last Supper\"<\/code>,<\/code><\/div>\n
            <\/code>\"type\"<\/code>: <\/code>\"Fresco\"<\/code><\/div>\n
        <\/code>}<\/code><\/div>\n
    <\/code>]<\/code><\/div>\n
}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Pretty-printing JSON and basic jq usage<\/h2>\n

jq is typically invoked by piping a piece of JSON to its standard input. For example:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n<\/td>\n
\n
\n
echo <\/code>'{ \"foo\" : \"bar\" }'<\/code> | jq<\/code><\/div>\n
{<\/code><\/div>\n
  <\/code>\"foo\"<\/code>: <\/code>\"bar\"<\/code><\/div>\n
}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Without any arguments jq simply outputs the JSON input data. Note that the output data has been reformatted. jq outputs pretty-printed JSON by default. This lets us pipe minimized JSON to jq and get a nicely formatted output.<\/p>\n

jq accepts one or more filter(s) as parameter. The simplest filter is . which returns the whole JSON document. So this example produces the same output as the previous example:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n<\/td>\n
\n
\n
echo <\/code>'{ \"foo\" : \"bar\" }'<\/code> | jq <\/code>'.'<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

We can now add a simple object identifier to the filter. For this we will use the previously mentioned artist.json<\/em> file. With .name we select the value of the name element:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'.name'<\/code><\/div>\n
\"Leonardo da Vinci\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Arrays can be navigated using the [] syntax:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'.artworks[0]'<\/code><\/div>\n
{<\/code><\/div>\n
  <\/code>\"name\"<\/code>: <\/code>\"Mona Lisa\"<\/code>,<\/code><\/div>\n
  <\/code>\"type\"<\/code>: <\/code>\"Painting\"<\/code><\/div>\n
}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

To get the name of the first painting we use:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'.artworks[0].name'<\/code><\/div>\n
\"Mona Lisa\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

If we want to get the names of all artworks we simply skip the array index parameter:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'.artworks[].name'<\/code><\/div>\n
\"Mona Lisa\"<\/code><\/div>\n
\"The Last Supper\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Processing curl and wget responses<\/h2>\n

Of course we can also pipe responses from remote systems to jq. This is not a specific feature of jq, but because this is a common use-case we look into two short examples. For these examples we will use the public GitHub API to get information about my blog-examples<\/a> repository.

<\/div> <\/div><\/p>\n

With curl this is very simple. This extracts the name and full_name properties from the GitHub API response:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n<\/td>\n
\n
\n
curl https:<\/code>\/\/api.github.com\/repos\/mscharhag\/blog-examples | jq '.name,.full_name'<\/code><\/div>\n
\"blog-examples\"<\/code><\/div>\n
\"mscharhag\/blog-examples\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Note we used a comma here to separate different two different filters.<\/p>\n

With wget we need to add a few parameters to get the output in the right format:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
wget -cq https:<\/code>\/\/api.github.com\/repos\/mscharhag\/blog-examples -O - | jq '.owner.html_url'<\/code><\/div>\n
\"https:\/\/github.com\/mscharhag<\/a>\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Pipes, functions and operators<\/h2>\n

In this section we will into a more ways of filtering JSON data.<\/p>\n

With the | operator we can combine two filters. It works similar as the standard unix shell pipe. The output of the filter on the left is passed to the one on the right.<\/p>\n

Note that .foo.bar is the same as .foo | .bar (the JSON element .foo is passed to the second filter which then selects .bar).<\/p>\n

Pipes can be combined with functions. For example we can use the keys function to get the keys of an JSON object:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'. | keys'<\/code><\/div>\n
[<\/code><\/div>\n
  <\/code>\"artworks\"<\/code>,<\/code><\/div>\n
  <\/code>\"name\"<\/code><\/div>\n
]<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

With the length function we can get the number of elements in an array:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
cat artist.json | jq <\/code>'.artworks | length'<\/code><\/div>\n
2<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

The output of the length function depends on the input element:<\/p>\n