Linux Shell Commands - 04 Pipes

This post is Part Four of the summary of some common Linux Shell Commands.

Pipes

Pipes connect the output and input.

Three I/O Stream

stdin: Input stream. File descriptor (FD) is 0.

• FD is an index.
• Kernel uses FD to visit files.
• FD are non-negative integers. Kernel returns a FD when open / create a new file. Read / write files also need FD.

stout: FD is 1

stderr: FD is 2

Redirection

Re. Input: [n] < file

Re. Output: [n] > file

• Note: This creates new file if file doesn’t exist, overwrites original content if file exists.
• e.g. cat file1 | cat > file2: Stdin is file1’s content, redirected to file2.

Re. Error: <command 1> |& <command 2>

• e.g. ls file1 |& cat > file2

• e.g. cat file 2 > err.log 1 > info.log: Stderr (2), stdout (1), save output and error in 2 separate files.

tee: Read stdin, and write to stdout & files.

• tee -a: append, not overwrite

• tee file1 file2: Write file1 to file2

• ls | tee file1: Write result to file1

• Quit tee: Ctrl + d

Text Processing

Sort

• sort -u: Get rid of repated lines

• sort -t: Split chars. e.g. Whitespace

• sort -o: Output to file

• sort -k: Sort according to one column

• sort -r: Sort, desceding (Default is ascending)

• e.g. sort -k 3 -o res.txt file1: Use file1’s data, sort according to 3rd column, output write to res.txt.

Combine

• paste file1 file2: Combine as one file

• join file1 file2: Find intersection (set)

Transform

• Format: tr [option] set1 set2

• e.g. echo "hello world" | tr a-z A-Z: Transform to upper cases.

xargs

• Pass parameters, and read data from stdin.

• e.g. tail -5 \etc\hosts | xargs: Whitespace replaces \n.

• xargs -a: Read from file, not from stdin

• xargs -d: Self-defined delimiters

• xargs -n: Max number of parameters per line

echo "f1xf2xf3" | xargs -n 2 -d x

• Delimiter is x

• 2 parameters per line

• Result:

    f1 f2
    f3