LPIC - 104 Manage Files

Objective

# File Management Commands

1 - File Commands

File naming

Linux filename length depends on filesystem

For ext4, limit is 255 char
.xx files are hidden : store configuration files

Special Chars : Do not use for file names

* - matches 0 ~ N chars (b*k)
? - matches single char (b?k)
[] - matches any char in the set (b[ao][a-z]k)
/ - folder
\ - folder in windows
"
^ - negate selection (i.e. except the selected one)

File Operation

Quickly determine file type: file (e.g. file blog.txt)

ls

The number after rwxr-xr-x in ls -l: Hard Link count 📌

# ls
--color  differentiate dirs, links, etc.
-R       recursive listing

-d       only dir name
-F       show file type
-i       display inode

Examples

# long list all dirs matches a*
ls -dl a*

# file globbing
ls b??k

# bracketed wildcard
ls b[a-z][^eio]t	# negate e

cp

📌 Use cp -a instead of cp -r

# cp
-a	recursive copy + keep all original attributes

-p	preserve ownership & permission
-r	recursive
-u	update, only if source is newer than dest

touch

Mainly for updating file’s timestamps

Linux maintain 3 timestamps for each file

Last file modified time
Last inode change time
Last access time

By default, touch set modified & access time to current time

-a	change access time (atime)
-m	change modified time (mtime)
-c	do not create new file

-r	set ref file, to replicate timestamp
-t	set specified time
    MMDDhhmm[[CC]YY][.ss]

Hard & Soft Links

Link: Give a file multiple identities

Hard link
Soft link (symbolic)

ln

ln creates hard link by default

# format
ln [option] source link

# options
-s	soft link

-f	remove all existing links
-i	interactive
-d	directory hard link (not supported by most filesystem)

Hard links

Points to same data on disk

Two entries that point to the same inode: a pseudo copy of a file, without truly copying the data
All hard links need to exist on a single low-level filesystem (cannot create hard links across filesystems)
To delete file: need to delete all hard links to the file
Can use hard link for file backup (the backup points to the data on disk)

Soft links

Points to original file (alias): If linked file is removed, soft link will be broken

Special file types: A separate file that points to the linked file / directory by name
Do not share same inode numbers, because do not point to the same disk data
Can point across filesystems 📌
Slower than hard links (but very tiny difference)

Other options

# remove link
unlink xx

# find the final linked file
readlink -f [soft linked file]

Stale links can be serious security problem.
Use soft link with caution, and remember to unlink.

Directory Commands

mkdir

# mkdir
-m [mode]	New directory has specific permission mode
-p	create parent directory
-v	verification (success message)

mkdir -v for verification (also rm, mv, etc.)

rmdir

1
2
3

# rmdir
-p	delete entire directory tree
-r	recursive

Examples

# remove empty directory only, with verification
rm -dv xx	

# delete directory by inode
ls -li
find . -inum 31162147 -exec rm -rv {} \;

2 - File Archiving

gzip, bzip2, xz

tar

cpio

dd

Compression

Compression tool ( gzip, bzip2, xz )

Apply compression to tarball as a whole (tar.gz)
Reduce tarball size, compared to compress file individually
View content of compressed file: Temporarily decompress, and show content to terminal stdout

Comparison:

gzip - oldest, least compression
bzip2 - improved compression
xz - newest, best compression (LZMA2 compression algorithm)
zip - normally we don’t use this one

Compress	Extension	Uncompress	View Content
`gzip`	`.gz` / `.tgz`	`gunzip -c`	`gzcat`
`bzip2`	`.bz2` / `.tbz`	`bunzip2 -c`	`bzcat`
`xz`	`.xz` / `.txz`	`unxz -b`	`xzcat`
`zip`	`.zip`	`unzip`	`zcat`

Example

# compress
bzip2 [xx]

# uncompress
unxz -b [xx]

# specify compression level
gzip -6 [xx]

Compression level: 1 to 9

1 - Fatest, but lowest compression
9 - Slowest, highest compression
6 - default compression level

tar

Tape Archiver: directs the output straight to tape device / regular file (No intermediate storage)

Tarballs: archives created by tar, compressed by gzip / bzip2

Distribute source code

.snar: tarball snapshot file

1 2	# create incremental backup with -g tar -g hello.snar -Jcvf hello_new.txz hello.txt

Options

General

-x	extract
-c	create archive
-v	verbose
-f	use [file] as archive file
-X	exclude files from archive (--exclude-from-file)

Archive options

1
2
3

-z	archive with gzip
-j	archive with bzip2
-J	archive with xz

Append

-A	append tar file to archive
-r	append non-tar file to archive
-u	append newer ones only (update)
-g	incremental / full archive based on metadata

Verification

1
2
3

-d	compare archive to file on disk (diff)
-t	list content
-W	veirify each file

Example

Compress

tar  -cvf xx.tar.gz  [xx folder, file]
tar -zcvf xx.tar.tgz [xx folder, file]
tar -jcvf xx.tar.tbz [xx folder, file]
tar -Jcvf xx.tar.txz [xx folder, file]

Extract

tar  -xvf xx.tar.gz
tar -zxvf xx.tar.tgz
tar -jxvf xx.tar.tbz
tar -Jxvf xx.tar.txz

Other

# list tar content
tar -tf xx.tar.gz 

# compare tarball memebrs to external files
tar -df hello.tgz

# auto verify backup file
tar -Wcvf hello.tar hello_*.txt

Cannot use -W if tarball is compressed: Verify first, compress later.

cpio

cpio = copy in and out

Restore data: read directly from tape device file

3 operating mode:

Copy-out (-o / --create) : Create archive & copy files
Coyp-in (-i / --extract) : Extract data from existing archive
Copy-pass (-p / --pass-through) : copy-out + copy-in. Copy directory from one location to another.

Options

-A	append to exsiting archive
-F xx	set cpio archive file in xx name
-t	list contents
-v	verbose

Examples

Archive directory: pass a list of files using standard input

# archive & compress
find ~/repo | cpio -o | xz > /mnt/data/repo.cpio.xz

# uncompress & unarchive
gunzip -c /mnt/data/repo.cpio.xz | cpio -i

dd

Archive filesystem / partition at low level

Create exact backup of entire partition

# back up partition
dd if=/dev/sda1 of=/mnt/data

# create empty file of particular size
# 1024 * 720B = 720 KB
dd if=/dev/zero of=file.img bs=1024 count=720

if - input size
of - output size
bs - block size (in bytes, default is 512)
count - length
status - level of info to be displayed
- none - error message only
- noxfer - no final transfer stats
- progress - periodic transfer stats

After block file is created, can use mkfs to create filesystem.

Zero an entire disk

1 2	# zero the disk dd if=/dev/zero of=/dev/sda1 status=progress

Need to perform at least 10 times to thoroughly wipe the disk.

Can also use /dev/random, /dev/urandom to write random data.

3 - Disk Quotas

Limit how many files / disk space a single user may consume

Modify quota in /etc/fstab

# modify
/dev/sda1	/home	ext4	usrquota,grpquota	1	1

# remount
mount -o remount /mnt/data

quotacheck - check quota
edquota - modify user quota, or temporarily in /etc/quotatab
quotaon - run from SysV startup script (installed by the distro’s quota package)

# File Ownership & Permissions

1 - Ownership

Three tiers of permissions

u - Owner
g - Group
o - World

chown & chgrp

# chown
chown user:group file.txt

# recursive
-R

# chgrp
chgrp -R group file.txt

2 - Manage Access

Permissions

1	-rwxr-xr-x file

First char: file type code
read - 4
write - 2
execute - 1

File type code	Meaning
`-`	Normal file
`d`	Directory
`l`	Soft link
`p`	Pipe
`s`	Socket (similar to pipe, but permits network & bi-directional links)
`b`	Block device (e.g. hard disks)
`c`	Character device (data is traferred in one-byte unit. e.g. parallel port, audio device)

When a directory’s execute bit is 1, meansthat its contents are searchable

Root can read / write any file, including those has 000 permission

Special permission bit

SUID & SGID (especially SUID root) programs are potential security risks.

SUID - 4

Set user ID: run program with file owner’s permission
1
2
# owner's execute bit: set to s
rwsr-xr-x 4755

SGID - 2

Set grop ID: run with file group permission

1
2
3

# group's execute bit: set to s
rwxr-sr-x	2755
rws-r-s---	6750

Sticky - 1

Protect files from being deleted by non file owners
1
2
# world's execute bit: set to t
rwxr-xr-t 1755

Modify File Mode

Files permissions are stored as part of the file’s inode, which isn’t part of the directory entry.

Read / write access to the directory or file doesn’t grant the right to change inode sturctures.

chmod

Octal mode: set specific absolute permission
Symbolic mode: make simple change (+, -, =)

# add execute to all
chmod a+x

# assign only read + write to owner & group
chmod ug=rw

# make owner and group permission equal
chmod g=u

# remove write from group, read + write from world
chmod g-w, o-rw

# add SUID + SGID + sticky
chomd u+s, g+s, o+t

Set Default Mode

umask: Bitwise removal from directory’s 777 permissions, or file’s 666 permissions

`umask`	Files	Directories
`000`	666	777
`002`	664	775
`027`	640	750
`077`	600	700
`277`	400	500

By default, file doesn’t have execution permission. So need to chmod +x

Modify

umask value is normally set in /etc/profile at login time
Default umask: 0022
1
umask -S # symbolic: u=rwx,g=rx,o=rx

change default group

1 2	# -l: re-init environment newgrp -l merikanto

Modify File Attributes

chattr

# example
chattr +i / -i file

A	no access time update
a	append, disable write access except appending data
c	kernel compresses data written to file
i	immutable. File cannot be deleted / linked / renamed
j	journal all data written to file
s	secure deletion
t	no tail-merging

Secure deletion

Kernel zeros the data blocks

When deleting a file with rm, its directory entry is removed, and inode marked as available for recycling.

But the file’s data blocks aren’t erased.

# Locating Files

1 - General

whereis

1 2	# executable + man page whereis ls

Find program executables / config files + man page
Does not search user directories

which

1 2	# -a: list all complete paths which -a python3

List complete path for command (weak search tool)
Remove alias: unalias py3
diff: determine text file’s differences

type

1 2	# -t: shorten output type -t locate

How a command will be interpreted:

built-in
alias
external command (absolute directory reference)

2 - Find

Brute-force approach to find files: slow but flexible

Option	Meaning
`-name xx`	By file name
`-iname xx`	By file name, ignore case
`-perm xx`	By permission mode
`-size n`	By file size
`-maxdepth n`	By directory depth
`-inum n`	By inode
`-mmin n`	File data changed `n` mins ago
`-cmin n`	Files status changed `n` mins ago

Examples

Can specify one or more paths
Use " " to make matches more accurate

# find all c files in home
find /home -name "*.c"

# permission audit
find /usr/bin -perm /4600

Permission as /4600: / ignores other permissions (e.g. 600), make it strictly 4600

3 - Locate

Limitation

Simple tool, search only on filenames (But much faster than find)
Maintains a database that updates once daily: /var/lib/mlocate/mlocate.db

Manual update: sudo updatedb 📌
slocate: secure locate. Prevent users from seeing file names in directories they shouldn’t access

Options

By default, locate adds wildcard to the pattern: *xx*.
To find exact match, use "xx"

-A	all
-b	only file names
-c	number of matches
-q	quiet. No error messages