In the LNMP development process, we often configure the nginx, and we will set similar like this

1
2
3

upstream php-fpm {
    server php-fpm:9000;
}

and I want to know what does upstream here mean.

So basically it’s about dependency, the Nginx server depend on PHP server to return back data. So PHP server is upstream of Nginx.

The Browser is depending on the Nginx server to return back data, so Nginx server is upstream of Browser.

On the other hand, Browser is downstream of Nginx server, because the data from Nginx will flow into the Browser, same as PHP, DB.

A production process is very similar to a river, so it’s easy to grasp that as the process goes from one step to the next, we’re moving downstream.

We can deduct the following rules:

• Dependency Rule: each item depends on all the items upstream from its viewpoint
• Value Rule: moving downstream, each step adds more value to the product

Does Docker build –no-cache actually download and refresh the base image

The –no-cache option will rebuild the image without using the local cached layers.

However, the FROM line will reuse the already pulled base image if it exists on the build host (the from line itself may not be cached, but the image it pulls is).

If you want to pull the base image again, you can use the –pull option to the build command. E.g.

docker build --no-cache --pull -t new-image-name:latest .

check more from here

lsof list open files

1
2
3
4

lsof -i -P -n | grep LISTEN
- -i selects the listing of all Internet and x.25 (HP-UX) network files
- -P inhibit the conversion of port numbers to port names
- -n inhibit the conversion of network numbers to host names

-P and -n will make the lookup faster

Example

List only files that listen to port 80

lsof -i:80 -P -n | grep LISTEN

What does 10064 mean in Git?

When you do git commit command, you probably have seen this information

1
2

 delete mode 100644 layouts/resource/index.html
 create mode 100644 layouts/resource/section.html

but what does this series of number mean?

The values shown are the 16-bit file modes as stored by Git, you can check here to know more:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

  32-bit mode, split into (high to low bits)

    4-bit object type
      valid values in binary are 1000 (regular file), 1010 (symbolic link) and 1110 (gitlink)
    
    3-bit unused

    9-bit unix permission. Only 0755 and 0644 are valid for regular files.
    
    Symbolic links and gitlinks have value 0 in this field.

If the file doesn’t mention directories; they are represented using object type 0100.

Each digit in the six-digit value is in octal, representing three bits; 16 bits thus need six digits, the first of which only represents one bit:

1
2
3
4
5
6
7

Type|---|Perm bits

1000 000 111101101
1 0   0   7  5  5

1000 000 110100100
1 0   0   6  4  4

Git doesn’t store arbitrary modes, only a subset of the values are allowed, from the usual POSIX types and modes (in octal, 12 for a symbolic link, 10 for a regular file, 04 for a directory) to which git adds 16 for Git links. The mode is appended, using four octal digits.

For files,

you’ll only ever see 100755 or 100644 (although 100664 is also technically possible);

directories are 040000 (permissions are ignored),

symbolic links 120000.

The set-user-ID, set-group-ID and sticky bits aren’t supported at all (they would be stored in the unused bits).

Problem

We have a list of IDs generated into a file F with this format:

1
2
3

2f9921bc-a4c9-419a-b069-f8f23c7727f7
26bf5cc3-73de-425e-9a6b-b70223387314
30ac5b4c-f4c6-42c6-901d-e8bdf16464b8

We want to generate a JSON object from the IDs with this structure:

1
2
3
4
5
6
7
8
9

{
    "users": [
        {
            "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
        }
    ],
    "organisation_id": 123,
    "user-type": "test"
}

The way we do it

we use jq

• what is jq
  • jq is a lightweight and flexible command-line JSON processor. 轻量级的命令行JSON处理器，非常灵活
• what are the main features
  • jq is like sed for JSON data
    • 你可以方便的使用它对结构化的数据进行slice and filter and map and transform
  • jq is written in portable C, and it has zero runtime dependencies.
    • 它没有运行时的依赖
• Tutorial
• Manual or you can run man jq

How do we do it

the following is the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    USER_IDS=$(jq -Rn '
        def assemble_user($user_id): {
            "user_id": $user_id
        };
        [
            inputs 
            | select(length>0)
            | assemble_user(.)
        ]' <<< "$USER_IDS")

    USER_JSON=$( jq -n \
                  --argjson user_ids "$USER_IDS" \
                  --arg organisation_id "$ORGANISATION_ID" \
                  --arg user_type "$USER_TYPE" \
                  '{organisation_id: $organisation_id, user_type: $user_type, users: $user_ids}' )

    echo "$USER_JSON"

Syntax of jq

• –raw-input/-R: Don’t parse the input as JSON. Instead, each line of text is passed to the filter as a string. If combined with –slurp, then the entire input is passed to the filter as a single long string.

• –null-input/-n: Don’t read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch.

• input and inputs, that read from the same sources (e.g., stdin, files named on the command-line) as jq itself.

  • input Outputs one new input.
  • inputs Outputs all remaining inputs, one by one.

Explaination of the process

• -R means jq will treat the anything it gets as a string not a JSON. if you don’t use -R and pass a string as input, it will throw error.

• -n means jq will not read any input. so it can work together with inputs to construct a JSON data.

  • if you do this echo '1\n2\n3' | jq -R '[inputs]' you will only get ["2","3"]. why? because without -n jq will read a input, which is 1, then the inputs will ouputs the remaining inputs, which are 2\n3.
  • if you do this echo '1\n2\n3' | jq -Rn '[inputs]', you will get ["1","2","3"]. why? because with -n jq will not read a input, so the inputs will outputs the remaining inputs, which are 1\n2\n3.

• select(length>0) for each input, it will filter out the empty input

• assemble_user(.) for each input, it will call the def assemble_user

• def assemble_user($user_id): {"user_id": $user_id} for each input like 123, it will return a object like {"user_id": 123}

• <<< this symbol means Here string

  • <<< is known as here-string. Instead of typing in text, you give a pre-made string of text to a program. For example, with such program as bc we can do bc «< 54 to just get output for that specific case, no need to run bc interactively. Think of it as the equivalent of echo ‘54’ | bc.

  check the difference between <,<<,<<< here

• --argjson means it has a JSON data as the argument.

• --arg means it has a string data as the argument

• jq -n --argjson user_ids "$USER_IDS" --arg organisation_id "$ORGANISATION_ID" you can structure a JSON data easily.

The result

• So first step I get the array with objects.
  • if you do this quickly

1
2
3
4
5
6
7

echo '2f9921bc-a4c9-419a-b069-f8f23c7727f7\n26bf5cc3-73de-425e-9a6b-b70223387314\n30ac5b4c-f4c6-42c6-901d-e8bdf16464b8' | jq -Rn '
        def assemble_users($user_id): {"user_id": $user_id};
        [
            inputs 
            | select(length>0)
            | assemble_users(.)
        ]'

you will get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[
  {
    "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
  },
  {
    "user_id": "26bf5cc3-73de-425e-9a6b-b70223387314"
  },
  {
    "user_id": "30ac5b4c-f4c6-42c6-901d-e8bdf16464b8"
  }
]

• Second, I assemble them together as a object. you will get

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
    "users": [
        {
            "user_id": "2f9921bc-a4c9-419a-b069-f8f23c7727f7"
        },
        {
            "user_id": "26bf5cc3-73de-425e-9a6b-b70223387314"
        },
        {
            "user_id": "30ac5b4c-f4c6-42c6-901d-e8bdf16464b8"
        }
    ],
    "organisation_id": 123,
    "user-type": "test"
}