Problem

We have a list of IDs generated into a file F with this format:

1
2
3

2f9921bc-a4c9-419a-b069-f8f23c7727f7
26bf5cc3-73de-425e-9a6b-b70223387314
30ac5b4c-f4c6-42c6-901d-e8bdf16464b8

We want to generate a JSON object from the IDs with this structure:

1
2
3
4
5
6
7
8
9

{
    "users": [
        {
            "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
        }
    ],
    "organisation_id": 123,
    "user-type": "test"
}

The way we do it

we use jq

  • what is jq
    • jq is a lightweight and flexible command-line JSON processor. 轻量级的命令行JSON处理器，非常灵活
  • what are the main features
    • jq is like sed for JSON data
      • 你可以方便的使用它对结构化的数据进行slice and filter and map and transform
    • jq is written in portable C, and it has zero runtime dependencies.
      • 它没有运行时的依赖
  • Tutorial
  • Manual or you can run man jq

How do we do it

the following is the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    USER_IDS=$(jq -Rn '
        def assemble_user($user_id): {
            "user_id": $user_id
        };
        [
            inputs 
            | select(length>0)
            | assemble_user(.)
        ]' <<< "$USER_IDS")

    USER_JSON=$( jq -n \
                  --argjson user_ids "$USER_IDS" \
                  --arg organisation_id "$ORGANISATION_ID" \
                  --arg user_type "$USER_TYPE" \
                  '{organisation_id: $organisation_id, user_type: $user_type, users: $user_ids}' )

    echo "$USER_JSON"

Syntax of jq

  • –raw-input/-R: Don’t parse the input as JSON. Instead, each line of text is passed to the filter as a string. If combined with –slurp, then the entire input is passed to the filter as a single long string.

  • –null-input/-n: Don’t read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch.

  • input and inputs, that read from the same sources (e.g., stdin, files named on the command-line) as jq itself.

    • input Outputs one new input.
    • inputs Outputs all remaining inputs, one by one.

Explaination of the process

  • -R means jq will treat the anything it gets as a string not a JSON. if you don’t use -R and pass a string as input, it will throw error.

  • -n means jq will not read any input. so it can work together with inputs to construct a JSON data.

    • if you do this echo '1\n2\n3' | jq -R '[inputs]' you will only get ["2","3"]. why? because without -n jq will read a input, which is 1, then the inputs will ouputs the remaining inputs, which are 2\n3.
    • if you do this echo '1\n2\n3' | jq -Rn '[inputs]', you will get ["1","2","3"]. why? because with -n jq will not read a input, so the inputs will outputs the remaining inputs, which are 1\n2\n3.

  • select(length>0) for each input, it will filter out the empty input

  • assemble_user(.) for each input, it will call the def assemble_user

  • def assemble_user($user_id): {"user_id": $user_id} for each input like 123, it will return a object like {"user_id": 123}

  • <<< this symbol means Here string

    • <<< is known as here-string. Instead of typing in text, you give a pre-made string of text to a program. For example, with such program as bc we can do bc «< 54 to just get output for that specific case, no need to run bc interactively. Think of it as the equivalent of echo ‘54’ | bc.

    check the difference between <,<<,<<< here

  • --argjson means it has a JSON data as the argument.

  • --arg means it has a string data as the argument

  • jq -n --argjson user_ids "$USER_IDS" --arg organisation_id "$ORGANISATION_ID" you can structure a JSON data easily.

The result

  • So first step I get the array with objects.
    • if you do this quickly
1
2
3
4
5
6
7

echo '2f9921bc-a4c9-419a-b069-f8f23c7727f7\n26bf5cc3-73de-425e-9a6b-b70223387314\n30ac5b4c-f4c6-42c6-901d-e8bdf16464b8' | jq -Rn '
        def assemble_users($user_id): {"user_id": $user_id};
        [
            inputs 
            | select(length>0)
            | assemble_users(.)
        ]'

you will get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[
  {
    "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
  },
  {
    "user_id": "26bf5cc3-73de-425e-9a6b-b70223387314"
  },
  {
    "user_id": "30ac5b4c-f4c6-42c6-901d-e8bdf16464b8"
  }
]
  • Second, I assemble them together as a object. you will get
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
    "users": [
        {
            "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
        },
        {
            "user_id": "26bf5cc3-73de-425e-9a6b-b70223387314"
        },
        {
            "user_id": "30ac5b4c-f4c6-42c6-901d-e8bdf16464b8"
        }
    ],
    "organisation_id": 123,
    "user-type": "test"
}