JQ

wiki https://en.wikipedia.org/wiki/Jq_(programming_language)
source (C) https://github.com/jqlang/jq/
basic manual https://jqlang.github.io/jq/manual/
home https://jqlang.github.io/jq/
repl online https://jqplay.org/
repl https://github.com/gpanders/ijq
awesome https://github.com/fiatjaf/awesome-jq
rust rewrite https://github.com/01mf02/jaq

language

#!/usr/bin/jq -f
  • jq -n # to quickly test snippets without input
  • jq sources ~/.jq automatically
  • First written in Haskell, but early rewritten in C for performance (?)
  • Created by Stephen Dolan in 2012
  • jq binary is of 33k
  • filters (aka functions)
    • all have an input and an output
    • so manual plumbing is not necessary
    • (aka to pass a value from one part of a program to the next)
    • eg: "add / length" both filters get the same input
  • variables are "lexically scoped bindings"
  • jq-mode
    • bug: wrong string interpolation can have (") inside (")
    • bug: wrong identation constructing objects

functions

  • the body consists of a SINGLE expression
  • single pass compiler (define a function before use it)
  • arity function overloading
  • has TCO for /0 arity functions ONLY
def funcname: expression; # takes stdin implicitly
def funcname(args) expression; # takes an argument explicitly
def A:
  def B(n): n + 1; # nested function definition
  B(10);

function arguments are filters (not values)

  • they act like what in other languages are callbacks

  • the filter expression gets passed into the function body as is 

    def add_mul(adder;multiplier):
  (. + adder) * multiplier;

add_mul(. + 5; . - 2); # => 200
# (. + . + 5) * . - 2
#  25         * 8

function arguments as values (syntatic sugar)

def my_func(arg):
  arg as $arg
  | #...
;

def my_func($arg):
  #....
;

recursion

def count_occurrences(x):
  if   length == 0 then # base
    0
  elif first  == x then # recursive
    (1 + (.[1:] | count_occurrences(x)))
  else # recursive
    (.[1:] | count_occurrences(x))
  end;

[11,22,33,22,44] | count_occurrences(22) # => 22

types

  • booleans
    • falsy values: null, false
type example
number 42, 3.14, 1e6, nan, infinite
string "hello"
boolean true, false
array [1, "2", {"foo": "bar"}]
object {"foo": 2}
null null
empty  

modules

import "MODNAME" as MODNAME;
def main:
  MODNAME::FUNCTION;
main

def FUNCTION:
    split(", ") ;

operators

operator description
+ addition, concatenation([]), merge({})
- subtraction
¦ pipe operator
¦= update operator (used for += -= *= /= %= //=)
, operator to join multiple streams
() to group subexpressions
[] constructor
{foo: .} constructor
+ string concatenation
+ arrays append
+ objects merge
A // B if A then A else B end (alternative operator)
? error suppresion, optional operator, shorthand for try
?// destructuring alternative operator
.[]? does NOT error when input is NOT an object or array
.foo? does NOT error when input is NOT an object

generators?

1 while(.<30; .*2) [1,2,4,8,16]  
1 repeat(.*2; error)? [2] repeats filter until error is raised
4 [.,1]¦until(.[0] < 1; [.[0]-1, .[1]*.[0]])¦.[1] 24  
  recurse???    

format & escape strings

@text just calls tostring
@json serializes input as JSON
@html applies HTML/XML escaping
@uri applies percent encoding
@csv rendered as CSV with double quotes
@tsv rendered as TSV
@sh escaped suitable for POSIX shell
@base64 as specified by RFC 4648

stdlib functions

general

filter description
debug like (.) but it prints to stderr too
range(TO)  
range(FROM;TO)  
range(FROM;TO;BY) produces a stream of numbers
empty returns empty
input outputs 1 NEW input
inputs outputs all remaining inputs, one by one
halt exit
halt_error/0 exit AND prints input
halt_error/1 exit with given code AND prints input
error/0 returns an error, can be catched (try/catch)
error/1 with message given

casting

[1,"1"] tonumber 1,1  
[1,"1",[1]] tostring "1","1","[1]"  
0 type "number"  

date

1425599507 todate "2015-03-05T23:51:47Z"
"2015-03-05T23:51:47Z" fromdate 1425599507
"2015-03-05T23:51:47Z" strptime("%Y-%m-%dT%H:%M:%SZ") [2015,2,5,23,51,47,4,63]
"2015-03-05T23:51:47Z" strptime("%Y-%m-%dT%H:%M:%SZ")¦mktime 1425599507
- now 1716057777.153488
1425599507 strftime("%H:%M:%S") "23:51:47"

stream

86, 99, 13 . + 1 87, 100, 14  
86, 99, 13 [ . + 1 ] [87], [100], [14]  

regex

input filter output description
"Hello World!" test("W") true to know if a substring matches the pattern
"Goodbye Mars" test("W") false  
  test(REGEX;FLAGS)    
  test([REGEX,FLAGS])    
  match([REGEX,FLAGS])    
"Hello World!" match("([aeiou])\\1") empty to extract the substring that matched
"Goodbye Mars" match("([aeiou])\\1") {"offset": 1, "length": 2, "string": "oo"  
    ,"captures": [  
    {"offset": 1  
    ,"length": 1  
    ,"string": "o"  
    ,"name": null }]}  
"Goodbye Mars" match("[aeiou]";"g") {"offset":1,"length":1,"string":"o","captures":[]}  
    {"offset":2,"length":1,"string":"o","captures":[]}  
    {"offset":6,"length":1,"string":"e","captures":[]}  
    {"offset":9,"length":1,"string":"a","captures":[]}  
"JIRAISSUE-1234" capture("(?<project>\\w+)-(?<issue_num>\\d+)") {"project":"JIRAISSUE","issue_num":"1234"} object of named captures
  capture(REGEX)    
  capture(REGEX;FLAGS)    
  capture([REGEX,FLAGS])    
"Goodbye Mars" scan("[aeiou]") "o", "o", "e", "a" only substrings, like match(RE,"g")
"Goodbye Mars" [scan("[aeiou]")] ["o", "o", "e", "a"]  
"first second" split("\\s+"; "") ["first","second"]  
  split(REGEX; FLAGS)    
  sub(REGEX; REPLACEMENT)    
  sub(REGEX; REPLACEMENT; FLAGS)    
  gsub(REGEX; REPLACEMENT)    
  gsub(REGEX; REPLACEMENT; FLAGS)    
"this: gnu, csv" gsub("\\b(?<tla>[[:alpha:]{3})\\b") "this: GNU, CSV"  
  ; "\(.tla ¦ ascii_upcase)")    

array

in filter out
{foo:[1],bar:[2]} .foo + .bar [1,2]
{foo:[1,2],bar:[2,3]} .foo - .bar [1]
[2,4,6,8][]   2,4,6,8
[2,4,6,8] .[] 2,4,6,8
[2,4,6,8] . + [1] [2,4,6,8,1]
[2,4,6,8] [ .[] + 1 ] [3,5,7,9]
[2,4,6,8] .[] + 1 3,5,7,9
[2,4,6,8] .[1] 4
[2,4,6,8] .[1+1] 6
[2,4,6,8] .[1:2] [4]
[2,4,6,8] limit(2;.[]) 2,4
[2,4,6,8] first 2
[2,4,6,8] last 8
[2,4,6,8] length 4
[2,4,6,8] indices(8) [3]
[2,4,6,8] contains([2]) true
[2,4,6,8] index(6) 2
[2,2,4,6,8,4] unique [2,4,6,8]
[{"foo":1},{"foo":1}] unique_by(.foo) [{"foo":1}]
["foo","bar","bazinga"] unique_by(length) ["foo","bazinga"]
[2,4,6,8] reverse [8,6,4,2]
[8,4,6,2] sort [2,4,6,8]
[2,4,6,8] min 2
[2,4,6,8] max 8
[2,4,6,8] add 20
["foo","bar"] add "foobar"
[{foo: 1, bar: 2}] add {foo: 1, bar: 2}
[72,101,108,108,111,33] implode "Hello!"
[2,4,6,8] nth(2) 6
{name: "Jane", age: 21} map(.age += 1) {name: "Jane", age: 22}
[2,4,6,8] map(. * 10) [20,40,60,80]
[2,4,6,8] [ .[] ¦ . * 10 ] equivalent to map()
["foo","bar"] add // "" "foobar"
["foo","bar"] join(",") "foo,bar"
{name: "Jane", age: 1} select(.age > 18) []
[2,[],3,[4],5] flatten [2,3,4,5]
[true,false] any true
[true,false] all false
[1,2,3,4] any(. >= 4) true
[1,2,3,4] all(. >= 4) false
[1] to_entries [{key:0,value:1}]
[10,20,30,40] keys [0,1,2,3]
[2,4] has(1) true
[2,4] has(4) false
1 in([0,23]) true
23 in([0,23]) false
[10,20] as [$foo,$bar] (empty, destructuring)
  reduce stream as $var (init;fn)  
[10,20,30,40] reduce .[] as $n (0; . + $n) 100
["A","B","C","D"] reduce .[] as $e ([]; [$e] + .) ["D","C","B","A"]
  • in the reduce fn
    • (.) is the accumulator
    • if you need to reduce the input, store it in a variable

string

being "Hello!" the INPUT

filter out description
ascii_downcase "hello!"  
ascii_upcase "HELLO!"  
/ "l" ["He","","o!"]  
split("l") ["He","","o!"]  
explode [72,101,108,108,111,33] splits into codepoints
implode    
startswith("!") false  
endswith("!") true  
test("He.*") true supports regex
contains("!") true  
inside("Hi, Hello!") true inverse of contains/1
length 6  
index("el") 1 position, otherwise null
+ "bar" "Hello!bar"  
* 3 "Hello!Hello!Hello"  
* 0 ""  
trim/ltrim/rtrim "Hello!" trims whitespace

math

https://jqlang.github.io/jq/manual/v1.7/#math

  • 1-input:
    • acos acosh asin asinh atan atanh cbrt ceil cos cosh erf erfc exp exp10 exp2 expm1 fabs floor gamma j0 j1 lgamma log log10 log1p log2 logb nearbyint pow10 rint round significand sin sinh sqrt tan tanh tgamma trunc y0 y1
    • pipe the input to the function
    • 1 | atan
  • 2-input:
    • atan2 copysign drem fdim fmax fmin fmod frexp hypot jn ldexp modf nextafter nexttoward pow remainder scalb scalbln yn
    • they ignore input
    • uses (;) to separate parameters
    • pow(2;10)
  • 3-input:
    • fma

object

  filter out  
{"a": 1, "b": 2, "c": 3} .[] 1,2,3 stream of values
{"a": 1, "b": 2, "c": 3} {a,c} {"a": 1, "c": 3}  
{a: {foo: 1}, b: {bar: 2}} .a + .b {foo: 1, bar: 2}  
{} .a null *projection, dot notation
{"a": 1, "b": 2} .a 1  
{"a": 1, "b": 2} . + {c: 3} {a: 1, b: 2, c: 3}  
{"a": 1, "b": 2} .["a"] 1  
{"a": 1, "b": 2} "foo" "foo"  
{"a": 1, "b": 2} .a = 100 {a: 100, b: 2} create/update property
{"a": 1, "b": 2} .a ¦= . + 100 {a: 101, b: 2}  
{"a": 1, "b": 2} {foo: .a} {foo: 1} new obj, old prop
{"name": "john"} as {name: $n} ¦ $n "john" as object destructuring
{"name": "john"} as {$name} ¦ $name "john" as object destructuring short
{"a": 1, "b": 2, "c": 3} flatten [1,2,3] array of values
{"a": 1, "b": 2, "c": 3} keys ["a","b","c"] array of keys
{"a": 1, "b": 2, "c": 3} keys_unsorted ["a","c","b"] array of keys
{"a": 1, "b": 2, "c": 3} has("a") true  
"a" in({"a": 2}) true  
{"a": 1, "b": 2, "c": 3} add 6 adds values
{"a": 1, "b": 2, "c": 3} del(.a) {"b":2, "c":3}  
{"a": 1, "b": 2, "c": 3} to_entries [{"key":"a","value":1},…]  
[{"key":"a","value":1}] from_entries {"a":1}  
{"Jane": 42} with_entries({key:(.value¦tostring),value:.key}) {"42": "Jane"}  
[{foo:1},{foo:2}] group_by(.foo) [[{foo:1}],[{foo:2}]]  
{"first": "jane"} .[] ¦= ascii_upcase {first: "JANE"}  
{"first": "jane"} map_values(ascii_upcase) {first: "JANE"} equivalent to above
  • with_entries(filter), is equivalent to: to_entries | map(filter) | from_entries

flags

    description
-n –null-input do NOT read input
-s –slurp reads whole STDIN as an array, runs script once
-c –compat-output minimizes output
  –unbuffered flush output after each json object is printed
-C –color-output  
-M –monochrome-output  
-f F –from-file FILE read .jq program from FILE
  –slurpfile VAR FILE reads json FILE into array variable $VAR
  –rawfile VAR FILE reads FILE raw into variable $VAR

snippets filters/expressions

filter description
.foo?  
.[] ¦ {msg: .commit.msg, name: .commit.commiter.name} builds a new json
path(..) ¦ map(tostring) ¦ join("/") instant schema
.parse.categories[].name the contents of each field's "name"
.["parse"] ¦ .["categories"] ¦ .[] ¦ .["name"] equivalent code using pipelines
input filter output description
  ..   recursive identity, from self to values
42 . 42 "identity"
99 42 42 "constant"
5 . * 2, . + 3, . / 5 10, 8, 1 multiple selectors?
"color" {(.): "red"} {color: "red"}  
{"k":1,"v":[8,9]} .v[1] 9  
{"k":1,"v":[8,9]} .v[ .["k"] ] 9  
[range(10)] map(select(. % 2 == 0)) [0,2,4,6,8]  
[range(10) ¦ select(. % 2 == 0) ] [0,2,4,6,8]  
[{}, true, {"a":1}] .[] ¦ .a? null, 1 optional operator
["1","invalid",4] .[] ¦ tonumber? 1, 4  
[86, 99, 13] length as $count 66 declaring variables with as
  ¦ add / $count    
{"size": 34.6 if ((.size¦floor)%2) == 0 "even" 1.6 - must have an ELSE
,"weight": 24.1} then "even"   1.7 - ELSE defaults to .
  else "odd"    
  end    
{"name":"John" "\(.name), Agent \(.nr)" "John, Agent 86" string interpolation
,"nr":"86"}      
{foo: 1} {bar: 2} [inputs] [{foo: 1}, {bar: 2}] streams to array

Created: 2023-11-27

Updated: 2024-11-16

Back