jq

Created: 27 Nov 2023
Updated: 17 Jan 2025
wiki https://en.wikipedia.org/wiki/Jq_(programming_language)
source (C) https://github.com/jqlang/jq/
basic manual https://jqlang.github.io/jq/manual/
home https://jqlang.github.io/jq/
repl online https://jqplay.org/
repl https://github.com/gpanders/ijq
awesome https://github.com/fiatjaf/awesome-jq
rust rewrite https://github.com/01mf02/jaq

cli

jq [OPTIONS] FILTER [INPUTFILES]
  <c> description
-n null-input do NOT read input (to quickly test scripts)
-s slurp reads whole STDIN as an array, runs script once
-c compat-output minimizes output
  unbuffered flush output after each json object is printed
-C color-output  
-M monochrome-output  
-f F from-file FILE read .jq program from FILE
  slurpfile VAR FILE reads json FILE into array variable $VAR
  rawfile VAR FILE reads FILE raw into variable $VAR

language

#!/usr/bin/jq -f
  • variables are "lexically scoped bindings"
  • jq-mode
    • bug: wrong string interpolation can have (") inside (")
    • bug: wrong identation constructing objects

filters / functions

  • all have an input, and an output
    • manual plumbing (passing of 1st argument) is NOT necessary
    • eg: "add / length" both filters get the same input
  • the body consists of a SINGLE expression
  • single pass compiler (define a function before use it)
  • arity function overloading
  • has TCO for /0 arity functions ONLY
def funcname: expression; # takes stdin implicitly
def funcname(args) expression; # takes an extra argument explicitly
def A:
  def B(n): n + 1; # nested function definition
  B(10);

function arguments are filters (not values)

  • they act like what in other languages are callbacks

  • the filter expression gets passed into the function body as is 

    def add_mul(adder;multiplier):
  (. + adder) * multiplier;

add_mul(. + 5; . - 2); # => 200
# (. + . + 5) * . - 2
#  25         * 8

function arguments as values (syntatic sugar)

def my_func(arg):
  arg as $arg
  | #...
;

def my_func($arg):
  #....
;

recursion

def count_occurrences(x):
  if   length == 0 then # base
    0
  elif first  == x then # recursive
    (1 + (.[1:] | count_occurrences(x)))
  else # recursive
    (.[1:] | count_occurrences(x))
  end;

[11,22,33,22,44] | count_occurrences(22) # => 22

types

  • booleans
    • falsy values: null, false
type example
number 42, 3.14, 1e6, nan, infinite
string "hello"
boolean true, false
array [1, "2", {"foo": "bar"}]
object {"foo": 2}
null null
empty  

modules

import "MODNAME" as MODNAME;
def main:
  MODNAME::FUNCTION;
main

def FUNCTION:
    split(", ") ;

operators

  description
+ addition, concatenation([]), merge({})
- subtraction
¦ pipe operator
¦= update operator (used for += -= *= /= %= //=)
, operator to join multiple streams
() to group subexpressions
[] constructor
{foo: .} constructor
+ string concatenation
+ arrays append
+ objects merge
A // B if A then A else B end (alternative operator)
? error suppresion, optional operator, shorthand for try
?// destructuring alternative operator
.[]? does NOT error when input is NOT an object or array
.foo? does NOT error when input is NOT an object

generators?

1 while(.<30; .*2) [1,2,4,8,16]  
1 repeat(.*2; error)? [2] repeats filter until error is raised
4 [.,1]¦until(.[0] < 1; [.[0]-1, .[1]*.[0]])¦.[1] 24  
  recurse???    

format & escape strings

@text just calls tostring
@json serializes input as JSON
@html applies HTML/XML escaping
@uri applies percent encoding
@csv rendered as CSV with double quotes
@tsv rendered as TSV
@sh escaped suitable for POSIX shell
@base64 as specified by RFC 4648

stdlib functions

general

  description
debug like (.) but it prints to stderr too
range(TO)  
range(FROM;TO)  
range(FROM;TO;BY) produces a stream of numbers
empty returns empty
input outputs 1 NEW input
inputs outputs all remaining inputs, one by one
halt exit
halt_error/0 exit AND prints input
halt_error/1 exit with given code AND prints input
error/0 returns an error, can be catched (try/catch)
error/1 with message given

casting

[1,"1"] tonumber 1,1  
[1,"1",[1]] tostring "1","1","[1]"  
0 type "number"  

date

1425599507 todate "2015-03-05T23:51:47Z"
"2015-03-05T23:51:47Z" fromdate 1425599507
"2015-03-05T23:51:47Z" strptime("%Y-%m-%dT%H:%M:%SZ") [2015,2,5,23,51,47,4,63]
"2015-03-05T23:51:47Z" strptime("%Y-%m-%dT%H:%M:%SZ")¦mktime 1425599507
- now 1716057777.153488
1425599507 strftime("%H:%M:%S") "23:51:47"

stream

86, 99, 13 . + 1 87, 100, 14  
86, 99, 13 [ . + 1 ] [87], [100], [14]  

regex

input filter output description
"Hello World!" test("W") true to know if a substring matches the pattern
"Goodbye Mars" test("W") false  
  test(REGEX;FLAGS)    
  test([REGEX,FLAGS])    
  match([REGEX,FLAGS])    
"Hello World!" match("([aeiou])\\1") empty to extract the substring that matched
"Goodbye Mars" match("([aeiou])\\1") {"offset": 1, "length": 2, "string": "oo"  
    ,"captures": [  
    {"offset": 1  
    ,"length": 1  
    ,"string": "o"  
    ,"name": null }]}  
"Goodbye Mars" match("[aeiou]";"g") {"offset":1,"length":1,"string":"o","captures":[]}  
    {"offset":2,"length":1,"string":"o","captures":[]}  
    {"offset":6,"length":1,"string":"e","captures":[]}  
    {"offset":9,"length":1,"string":"a","captures":[]}  
"JIRAISSUE-1234" capture("(?<project>\\w+)-(?<issue_num>\\d+)") {"project":"JIRAISSUE","issue_num":"1234"} object of named captures
  capture(REGEX)    
  capture(REGEX;FLAGS)    
  capture([REGEX,FLAGS])    
"Goodbye Mars" scan("[aeiou]") "o", "o", "e", "a" only substrings, like match(RE,"g")
"Goodbye Mars" [scan("[aeiou]")] ["o", "o", "e", "a"]  
"first second" split("\\s+"; "") ["first","second"]  
  split(REGEX; FLAGS)    
  sub(REGEX; REPLACEMENT)    
  sub(REGEX; REPLACEMENT; FLAGS)    
  gsub(REGEX; REPLACEMENT)    
  gsub(REGEX; REPLACEMENT; FLAGS)    
"this: gnu, csv" gsub("\\b(?<tla>[[:alpha:]{3})\\b") "this: GNU, CSV"  
  ; "\(.tla ¦ ascii_upcase)")    

array

in filter out
{foo:[1],bar:[2]} .foo + .bar [1,2]
{foo:[1,2],bar:[2,3]} .foo - .bar [1]
[2,4,6,8][]   2,4,6,8
[2,4,6,8] .[] 2,4,6,8
[2,4,6,8] . + [1] [2,4,6,8,1]
[2,4,6,8] [ .[] + 1 ] [3,5,7,9]
[2,4,6,8] .[] + 1 3,5,7,9
[2,4,6,8] .[1] 4
[2,4,6,8] .[1+1] 6
[2,4,6,8] .[1:2] [4]
[2,4,6,8] limit(2;.[]) 2,4
[2,4,6,8] first 2
[2,4,6,8] last 8
[2,4,6,8] length 4
[2,4,6,8] indices(8) [3]
[2,4,6,8] contains([2]) true
[2,4,6,8] index(6) 2
[2,2,4,6,8,4] unique [2,4,6,8]
[{"foo":1},{"foo":1}] unique_by(.foo) [{"foo":1}]
["foo","bar","bazinga"] unique_by(length) ["foo","bazinga"]
[2,4,6,8] reverse [8,6,4,2]
[8,4,6,2] sort [2,4,6,8]
[2,4,6,8] min 2
[2,4,6,8] max 8
[2,4,6,8] add 20
["foo","bar"] add "foobar"
[{foo: 1, bar: 2}] add {foo: 1, bar: 2}
[72,101,108,108,111,33] implode "Hello!"
[2,4,6,8] nth(2) 6
{name: "Jane", age: 21} map(.age += 1) {name: "Jane", age: 22}
[2,4,6,8] map(. * 10) [20,40,60,80]
[2,4,6,8] [ .[] ¦ . * 10 ] equivalent to map()
["foo","bar"] add // "" "foobar"
["foo","bar"] join(",") "foo,bar"
{name: "Jane", age: 1} select(.age > 18) []
[2,[],3,[4],5] flatten [2,3,4,5]
[true,false] any true
[true,false] all false
[1,2,3,4] any(. >= 4) true
[1,2,3,4] all(. >= 4) false
[1] to_entries [{key:0,value:1}]
[10,20,30,40] keys [0,1,2,3]
[2,4] has(1) true
[2,4] has(4) false
1 in([0,23]) true
23 in([0,23]) false
[10,20] as [$foo,$bar] (empty, destructuring)
  reduce stream as $var (init;fn)  
[10,20,30,40] reduce .[] as $n (0; . + $n) 100
["A","B","C","D"] reduce .[] as $e ([]; [$e] + .) ["D","C","B","A"]
  • in the reduce fn
    • (.) is the accumulator
    • if you need to reduce the input, store it in a variable

string

being "Hello!" the INPUT

filter out description
ascii_downcase "hello!"  
ascii_upcase "HELLO!"  
ltrimstr("Hell") "o!"  
rtrimstr("Hell") "Hello!"  
/ "l" ["He","","o!"]  
split("l") ["He","","o!"]  
explode [72,101,108,108,111,33] splits into codepoints
implode    
startswith("!") false  
endswith("!") true  
test("He.*") true supports regex
contains("!") true  
inside("Hi, Hello!") true inverse of contains/1
length 6  
index("el") 1 position, otherwise null
+ "bar" "Hello!bar"  
* 3 "Hello!Hello!Hello"  
* 0 ""  

math

https://jqlang.github.io/jq/manual/v1.7/#math

  • 1-input:
    • acos acosh asin asinh atan atanh cbrt ceil cos cosh erf erfc exp exp10 exp2 expm1 fabs floor gamma j0 j1 lgamma log log10 log1p log2 logb nearbyint pow10 rint round significand sin sinh sqrt tan tanh tgamma trunc y0 y1
    • pipe the input to the function
    • 1 | atan
  • 2-input:
    • atan2 copysign drem fdim fmax fmin fmod frexp hypot jn ldexp modf nextafter nexttoward pow remainder scalb scalbln yn
    • they ignore input
    • uses (;) to separate parameters
    • pow(2;10)
  • 3-input:
    • fma

object

in filter out  
{"a": 1, "b": 2, "c": 3} .[] 1,2,3 stream of values
{"a": 1, "b": 2, "c": 3} {a,c} {"a": 1, "c": 3}  
{a: {foo: 1}, b: {bar: 2}} .a + .b {foo: 1, bar: 2}  
{} .a null *projection, dot notation
{"a":1 , "b":2} .a 1  
{"a":1 , "b":2} . + {c: 3} {a: 1, b: 2, c: 3}  
{"a":1 , "b":2} .["a"] 1  
{"a":1 , "b":2} "foo" "foo"  
{"a":1 , "b":2} .a = 100 {a: 100, b: 2} create/update property
{"a":1 , "b":2} .a ¦= . + 100 {a: 101, b: 2}  
{"a":1 , "b":2} {foo: .a} {foo: 1} new obj, old prop
{"name": "john"} as {name: $n} ¦ $n "john" as object destructuring
{"name": "john"} as {$name} ¦ $name "john" as object destructuring short
{"a": 1, "b": 2, "c": 3} flatten [1,2,3] array of values
{"a": 1, "b": 2, "c": 3} keys ["a","b","c"] array of keys
{"a": 1, "b": 2, "c": 3} keys_unsorted ["a","c","b"] array of keys
{"a": 1, "b": 2, "c": 3} has("a") true  
"a" in({"a": 2}) true  
{"a": 1, "b": 2, "c": 3} add 6 adds values
{"a": 1, "b": 2, "c": 3} del(.a) {"b":2, "c":3}  
{"a": 1, "b": 2, "c": 3} to_entries [{"key":"a","value":1},…]  
[{"key":"a","value":1}] from_entries {"a":1}  
{"Jane": 42} with_entries({key:(.value¦tostring),value:.key}) {"42": "Jane"}  
[{foo:1},{foo:2}] group_by(.foo) [[{foo:1}],[{foo:2}]]  
{"first": "jane"} .[] ¦= ascii_upcase {first: "JANE"}  
{"first": "jane"} map_values(ascii_upcase) {first: "JANE"} equivalent to above
  • with_entries(filter), is equivalent to: to_entries | map(filter) | from_entries

snippets

filter description
.foo?  
.[] ¦ {msg: .commit.msg, name: .commit.commiter.name} builds a new json
path(..) ¦ map(tostring) ¦ join("/") instant schema
.parse.categories[].name the contents of each field's "name"
.["parse"] ¦ .["categories"] ¦ .[] ¦ .["name"] equivalent code using pipelines
in filter out description
  ..   recursive identity, from self to values
42 . 42 "identity"
99 42 42 "constant"
5 . * 2, . + 3, . / 5 10, 8, 1 multiple selectors?
"color" {(.): "red"} {color: "red"}  
{"k":1,"v":[8,9]} .v[1] 9  
{"k":1,"v":[8,9]} .v[ .["k"] ] 9  
[range(10)] map(select(. % 2 == 0)) [0,2,4,6,8]  
[range(10) ¦ select(. % 2 == 0) ] [0,2,4,6,8]  
[{}, true, {"a":1}] .[] ¦ .a? null, 1 optional operator
["1","invalid",4] .[] ¦ tonumber? 1, 4  
[86, 99, 13] length as $count 66 declaring variables with as
  ¦ add / $count    
{"size": 34.6 if ((.size¦floor)%2) == 0 "even" 1.6 - must have an ELSE
,"weight": 24.1} then "even"   1.7 - ELSE defaults to .
  else "odd"    
  end    
{"name":"John" "\(.name), Agent \(.nr)" "John, Agent 86" string interpolation
,"nr":"86"}      
{foo: 1} {bar: 2} [inputs] [{foo: 1}, {bar: 2}] streams to array