Merikanto

一簫一劍平生意，負盡狂名十五年

LPIC - 101 Command Line

Veröffentlicht am in
Symbole im Artikel gezählt: 4k Wörter Lesezeit ≈ 5 min

Objective


Work on the command line


Linux shell types

  • bash / bsh
  • tcsh / csh (c-shell)
  • ksh (korn shell)
  • zsh 
1
2
3
4
5
# which shell that /bin/sh points to
readlink /bin/sh 

# shell metacharacters
*  ?  '  "  ;  &  |  \  $  ^  [  ]  (  )  <  >

Note: Adjusting bash config files (~/.bashrc) only affects bash


Other notes

  • Directory structure: virtual directory, with one single base directory (root dir)

  • Check shell Built-in commands

    1
    2
    3
    4
    5
    type echo
    # echo is a shell builtin

    type uname
    # uname is /usr/bin/uname

  • Common env vars

    • Use set to display active env vars
    • Use unset to reverse edit
    • Use env or printenv to view env vars
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # default editor
    EDITOR

    # default screen-based editor (e.g. nano, vim)
    VISUAL

    # system hostname
    HOSTNAME

    # Primary cmdline prompt string
    PS1

    # current shell level
    SHLVL

    # user time zone
    TZ


PATH env var

  • root path should never include current dir (./)
  • Append new path to the end of path, so standard programs take precedence
  • Env var: modify program behavior

Shell command tricks

  • Ctrl+R: search command history (use ESC or Ctrl+G to exit)
  • Ctrl+ ←/→: move one word per time
  • Ctrl+K: delete cursor to end
  • Ctrl+X & Backspace: delete start to cursor
  • Ctrl+XE: open editor to edit commands ( export EDITOR=/usr/bin/vim )

Shell history

  • Clean history: history -c
  • Execute command from history: !200
  • View last 10 commands: history 10
  • Display without line number: history | cut -c 8-

Shell config

  • global: /etc/bash.bashrc
  • $TERM: xterm-256color
  • Display all env vars: env


Basic file editing

Many utilities do not change texts within a file, unless redirection (e.g. sed, cut)

They only display modified text to stdout


Message Digest (Integrity)

  • md5sum
  • sha256sum
  • sha512sum (best)

1 - Vim

Notes

  • vi editor was a Unix text editor. Vim is vi improved
  • 3 standard modes
    • Command mode (normal)
    • Insert mode (edit, entry)
    • Ex mode (colon commands)

Shortcuts

Guides: vimtutor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
^	line start
$	line end

gg	file start
G	file end

?	forward search
/	backward search

A	insert at line end
o	insert new line below cursor

:!	execute shell command & display results, but not quitting vim
:r	read file contents, include in editor buffer area


Ctrl + ZZ, :x,  :wq	save & quit


Streams, pipes, redirects


Redirection

  • >> : append (create new file if doesn’t exist)
  • 2> : new file with stderr
  • 2>> : append stderr
  • &> : new file with stdout & stderr
  • << : heredoc (terminate input: Ctrl+D)

A trick: Redirect stdout to /dev/null (get rid of data)

1
whine 2> /dev/null

  • tee 

    1
    2
    3
    4
    5
    6
    7
    8
    # Program:
    print("hello")

    # tee: store & view the output
    python hello.py | tee output.txt

    # append texts:
    python hello.py | tee -a output.txt

Pipe

xargs: build command from stdin

  • -d "\n" : xargs use both spaces& newlines as item delimiters

  • e.g.

    1
    2
    3
    4
    5
    6
    7
    8
    # Remove all files with ~
    find ./ -name "*~" | xargs -d "\n" rm

    # Equals: (`: separate command)
    rm `find ./ -name "*~"`

    # Equals: Shell expansion (命令嵌套，可嵌套多个)
    rm $(find ./ -name "*~")

    Output of above command is passed to rm, as if it’s typed in the shell



Process text streams with filters


1 - Combine files

  • cat
  • join
  • paste

cat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# tac: reverse lines
tac 1.txt 2.txt > combined.txt

# Use tee to view stdout & store
cat 1.txt 2.txt | tee combined.txt

# Number lines: (-n, --number) 
cat -n 1.txt 2.txt | tee -a combined.txt

# Delete number lines
cat 1.txt | cut -c 4-

# see file ending (LF, linefeed)
cat -E 1.txt
  • Number non-blank lines (-b, --number-nonblank)
  • Show end of file with $ (-E, --show-end)
  • Compress blank lines (-s, --squeeze-blank)

join

Need to have similarities of fields (columns) between file 1 & 2

1
2
3
4
5
6
# Note: files cannot have blank lines, else will display error:
# > join: 1.txt:11: is not sorted
join 1.txt 2.txt

# File a & b. Use 3rd column in 1, 4th column in 2
join -1 3 -2 4 a.txt b.txt

paste

1
2
# merge files line by line
paste 1.txt 2.txt


2 - Transform files

  • expand / unexpand
  • od
  • sort
  • split
  • tr
  • uniq

expand / unexpand

1
2
3
4
5
# Convert tabs to space
expand 1.txt -t 4

# Convert space to tabs
unexpand 1.txt --tabs=4

od (octal dump)

1
2
# Display file dump in hex
od -h 1.txt

sort

1
2
3
4
5
6
7
8
# Sort by field (-k, --key=)
sort 1.txt -k 2		# sort by 2nd column

# numeric sort, output to file, -o
sort -n -o new.txt old.txt

# numeric sort, output to file, redirection
sort -n old.txt > new.txt
  • ignore case (-f, --ignore-case)
  • Numeric sort (-n, --numeric-sort)
  • Reverse order (-r, --reverse)

uniq

1
2
# remove duplicate lines, use with sort
sort 1.txt | uniq

split

1
2
3
4
5
# split by bytes (-b, --bytes=)
split -b 30 1.txt 	 # 30 bytes each file, default prefix (xaa, xab, ...)

# split by lines (-l, --lines=)
split -l 4 1.txt N-	 # 4 lines each file, with prefix (N-aa, N-ab, ...)

tr

change individual chars from stdin

1
2
3
4
5
6
7
8
# change letter h, i to H, J
tr hi HJ < 1.txt 

# delete chars
tr -d h < 1.txt 	# delete h from file

# delete all numbers
tr -d [:digit:] < 1.txt


3 - Format files

  • fmt
  • nl
  • pr

fmt

1
2
# No more than 80 char wide
fmt 1.txt -w 80

nl

1
2
3
4
5
# Number all non-blank lines
nl 1.txt

# Number all lines (-b, --body-numbering)
nl -b a 1.txt		# style: all

pr

1
2
3
4
5
# prepare file for print
pr 1.txt

# print out, numbered, double-spaced, 50 lines per page
cat -n 1.txt | pr -dfl 50 | lpr


4 - View files

  • head / tail
  • more / less

head / tail

1
2
3
4
5
6
7
8
9
10
11
12
13
head -n 5
tail -n 10		# equals:  head -n -10

# cut chars in a pipe
head -c 5
tail -c 5

# stream logs (-f, --follow)
tail -f

# Extract portions with head + tail
# Display line 11 - 15
head -n 15 1.txt | tail -n 5


more / less

less is a better version of more

Basic:

  • Move forward: f
  • Move backward: b
  • Search forward: N
  • Search backward: Shift + N
  • Move to line 50: g50
  • Backward search mode: ? (N will search backward)
  • Help inside less: h

Visual mode: v (Edit current file with $EDITOR)

  • Exit visual mode: :q


5 - Summarize files

  • cut
  • wc

cut

  • By char: -c
  • By field: -f (a field is a tab-delimited section of a line)
  • Change delimiter: -d char
  • Numbers: can choose a range (cut -c 2-4)
1
2
3
4
5
6
7
8
9
10
11
# get ether address of wlan
ifconfig wlp2s0 | grep ether | cut -c 15-31	# 9c:b6:d0:9c:35:37


# cut selected char
echo "foobar" | cut -c 1,6	# fr


# field delimiter: use space ' ' to separate fields
cut -d ' ' -f1
cut -f1 '-d '  # same as above

wc

1
2
3
4
wc 1.txt
# 12  	 47 	257	1.txt
# lines, words, bytes 
# -l	 -w     -c


Search text files with regex

  • grep
    • Find matching text within a file & print out result
    • Cannot use grep to make changes
  • sed

Regex (basic & extended)

Quotation marks are not regex chars

  • ^ line start

  • $ line end

  • [] match any char inside

    b[aei]g –> bag, beg, big

  • - range

    a[2-4]z –> a2z, a3z, a4z

  • . any single char (except newline)

    a.z –> a2z, abz, aQz (and any other 3-char string that’s a.z)

  • * appears ≥ 0 times

    Use .* for substring match

  • + appears ≥ 1 times

  • ? 0 or 1 match

  • | multiple possible matches

  • () group expressions

  • \ escape char


Chracter classes

predefined names

1
2
3
4
5
6
7
8
9
10
[:alnum:]	# alpha & numeric
[:alpha:]
[:digit:]

[:lower:]
[:upper:]

[:blank:]	# blank char (e.g. tab)
[:space:]	# space char (e.g. space)
[:punct:]

1 - grep

  • Count matching lines -c, --count
  • Ignore case -i, --ignore-case
  • Recursive search -r, --recursive (or: rgrep)
  • Extended regex -E (or: egrep)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# use char class, double brackets
grep [[:digit:]] int.txt

# find all files that contain 0 or 1 in eth (rgrep, grep -r)
sudo rgrep eth[01] /etc/*

# grep OR
egrep 'E1 | E2'

# grep AND
egrep 'E1.*E2'
grep 'E1' | grep 'E2'

# grep NOT	(v means inverse grep)
grep -v 'E1'

Grep AND, OR, NOT: https://www.thegeekstuff.com/2011/10/grep-or-and-not-operators/


2 - sed

  • Directly modifies file content from stdin & send edited file to stdout (Stream editor)

  • Sed operates on addresses (line numbers)

    If no address: Operate on entire file

Usage

  • i\txt insert txt to file
  • c\txt replace with provided text
  • ADDRcTEXT see usage below
  • s/regex/replace substitute text that matches regex
  • pattern/d delete lines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# modify stdin	(donuts donuts)
echo "cake cake" | sed 's/cake/donuts/g'

# replace 1st occurrence of 2012 on each line with 2013
sed -i 's/2012/2013/' 1.txt > 2.txt

# global search (replace all, doesn't matter if it's 1st)
sed 's/2012/2013/g'

# delete 1st line contains cake
sed '/cake/d' cake.txt

# address is line number. Insert Hello at line 4
sed '4cHello' cake.txt

# use extended regex in script
sed -r / --regexp-extended

# use multiple scripts
sed -e 's/2012/2013/ ; s/cake/donuts/g' cake.txt

Slash (/) isn’t the only delimiter in sed.

You can use any character as a delimiter that’s not part of either string. (e.g. |, :)



Practice

exec: rest of command to replace current shell

Exit from program: xterm window will close


Limit of pipe: based on input buffer size


Text file record: single file line that ends in newline (ASCII char LF)


A file descriptor is a number that represents process’s open files


stdout goes to the current termianl (/dev/tty)