LPIC - 107 Linux System Admin

Objective

Manage User & Groups

Linux is a multi-user system that relies on accounts

Traditional Linux security control - DAC (Discretionary Access Control)

1 - Users

Summary

Commands
- getent - view account info
- chage - change account password setting
- passwd
- useradd
- usermod
- userdel
User account creation process 📌
- Input
  - /etc/default/useradd
  - /etc/login.defs - set account numbering limits
  - /etc/skel
- Output (changes)
  - /etc/passwd - account info
  - /etc/shadow
  - /etc/group

Notes

UID=0 - root user
First regular user account - UID is 500 or 1000
Reusing UID can cause problems, if old user’s files are not cleaned 📌

Attention when creating usernames

Via useradd - Forbid uppercase letters & most punctuation
Other utilities truncate usernames (longer than 8 char)
Usernames in Linux are case-sensitive
Safe practice: Usernames in all lower-case letters

Create user - useradd

Create user: Two steps
- Create user - useradd
- Set password - passwd
Copy files from skeleton directory - /etc/skel
- Provide core set of config files, which should be present in user’s home directory
Special accounts (shells)
- Some systesm include a shutdown account with default shell /sbin/shutdown
  Logging into this account immediately shuts down the system
- System service account - /sbin/nologin
- Logout - Account with shell /sbin/false, /sbin/logout

# create user merikanto
sudo useradd -m -g kk -G sigure, kk -s /bin/bash merikanto
sudo passwd merikanto

-D	# show file content  📌 

-m	# auto creates home dir
-d	# specify home dir
-g	# set group name / GID
-G	# set additional group
-s	# set default shell

Modify settings in /etc/default/useradd

1 2	# modify SHELL directive sudo useradd -D -s /bin/bash

When use useradd -G for additional groups:
Need to include all current groups. Any unlisted groups will be removed.

Solution: Use useradd -aG. Keep current membership: -a

Manage password & users - passwd

pwconv - Migrate password from /etc/passwd to /etc/shadow

# example
passwd merikanto

-S	# display account password status  📌 

-l	# lock account / temp suspension of user access
-u	# unlock account

View account records: getent

/etc/shadow - getent shadow [user]
/etc/passwd - getent passwd [user]
/etc/group - getent group [user]

View account password status

passwd -S (3 status)
- P - usable password
- NP - not usable
- L - locked
chage -l (more human readable)

Modify user - usermod

Change contents of /etc/passwd & /etc/shadow

# move user's file to new dir xx
usermod -md	

# change login name
useradd -l

If change account’s UID, need to manually update UID on all files

1	chown -R merikanto /home/merikanto

Modify account expiration - chage

-l	# display info

-d	# set last password change date
-E	# set expiration date

Modify account config files - /etc/passwd & /etc/shadow

Fields in /etc/passwd
- Username
- Password - x (shadow password, encrypted in /etc/shadow)
- UID
- Primary GID 📌
- Comment
- Home directory
- Default Shell
Permissions
- /etc/shadow has restrictive permissions - 600
- /etc/passwd is readable by normal users - 644

Network Account Databases (NAD)

Types
- NIS - Network Information System
- LDAP - Lightweight Directory Access Protocol
- AD - Active Directory
Enable at OS installation - Need to know server IP & protocol
Activate NAD
- Install relevant packages
- Edit /etc/nsswitch.conf
- Edit /etc/pam.d - PAM (Pluggable Authentication Module)

Delete users - userdel

# force remove all files from mail & user home dir
userdel -rf merikanto

# confirm file deletion
find / -uid xx

Attention: Samba

Samba keeps its own list of users
Delete user:
- Edit /etc/samba/sbmpasswd
- Command - smbpasswd -x merikanto

2 - Groups

Summary

Commands
- groupadd
- groupmod
- groupdel
- groups - audit account group membership
File Locations
- /etc/group
- /etc/gshadow

Notes

Set user’s primary group: /etc/passwd
User access other groups files:
- User is a group member
- The group has access permission
Run programs with a different group: switch group with newgrp
1
newgrp kk

Create group - groupadd

# most common
groupadd kk

-r	# create system group
-g	# set GID, override default gid set by system

Modify group - groupmod

Usually use usermod to add user to a group
Add users to group without specifying existing membership
- Edit /etc/group
- gpasswd
- usermod -aG ✅

Manage password & groups - gpasswd

# change group password
gpasswd kk

-a	# add user
-A	# add as group admin
-M	# bulk add users
-d	# delete user

Modify group config files - /etc/group & /etc/gshadow

Fields in /et/group
- Group name
- Password - x (shadow password, encrypted in /etc/gshadow)
- GID
- User list

Delete groups - groupdel

First check whether group is any user’s primary group
Can also leave orphaned files

# delete group
groupdel kk

# delete all related files
sudo find / -gid xx 2>/dev/null

System Logging

Provide unified means of handling log files.

1 - Syslog

Syslog Daemon - syslogd

syslogd runs in the background, waiting for events to trigger
Install - sysklogd package
- syslogd
- klogd - logging kernel messages
Alternatives
- syslog-ng - Supports advanced filtering
- metalog
- rsyslogd - Speed (rocket fast)
- systemd-journald

Config - /etc/syslog.conf

1	service.log_level location

Selector - service (facility) + log level (priority)
Valid codes for facility
- * - all facilities
- auth, security
- authpriv
- cron
- daemon - general system services
- kern
- lpr - printer
- mail
- mark - reserved for internal use
- news - news application
- syslog
- user
- uucp - Unix-to-Unix copy
- local0 - local7 - locally defined
Valid codes for priority
- * - all priorities
- ! - reverse (make the log level as highest)
- debug
- info
- notice
- warning
- err
- crit
- alert
- emerg

# send all emerg-level logs to all users
*.emerg    *

# save logs between info & err level to /var/log/kernel-info
kern.info;kern.!err    /var/log/kernel-info

rsyslogd

Config file - /etc/rsyslogd.conf
Actions
- Foward to regular file
- Pipe to application
- Display in terminal
- Send to remote host
- Send to list of users
- Send to all logged-in users

# log with specific level
kern.=crit

# - : not sync after each write
# .none = except (handle all events except security events)
*.*;auth,authpriv.none	-/var/log/syslog

# omusrmsg: send msg to user account
*.emerg	:omusrmsg:kk

Send log to remote server

Edit config file - /etc/rsyslogd.conf

Reload config file / restart rsyslogd after editing

# syntax
log  TCP|UDP[(z#)]HOST:[PORT#]

# example
*.*  @@(z9)merikanto.io:6514

@ - UDP, @@ - TCP
z - Use zlib to compress
# - Compression level (9 is the highest)
HOST - FQDN / IP

2 - Journal

Config file - /etc/systemd/journald.conf

Storage

Options - auto, persistent, volatile, none
auto - store logs in /var/log/journal/ if directory exists (log will persist through restart)
else store in /run/log/journal/ (temporary)
persistent - always create /var/log/journal/ and store logs
volatile - always temporary (/run/log/journal/)
none - all event messages are discarded

Layered logging

Journal client method - allow syslog protocol program to act as journal client, read entries stored in journal
For /etc/rsyslog.conf - Load with Modload
- imusock - local system logging
- imjournal - systemd journal
Forward to Syslog
- Edit /etc/systemd/journald.conf - Set ForwardToSyslog to yes
- Restart to reload config - systemctl restart systemd-journald

Journalctl

View journal entries
journald doesn’t store journal entries in text files, but in binary file format

Similar to database. Binary format for fast index and quick search
Format: journalctl [options] [matches]

Options

# last 10 lines, turn off pager
sudo journalctl -n 10 --no-pager

-a	# display all data fields
-e	# jump to end
-k	# only kernel entries
-r	# reverse the order
-f	# real-time log stream

-n [num]	# most recent number
-S [date]	# start from (2022-04-07:08:00:00)
-U [date]	# end with
-u [pattern]	# pattern match

Matches - Filter types of journal entries

PRIORITY
_HOSTNAME
_SYSTEMD_UNIT
_UDEV_SYSNAME - received from the specified device

1 2	# example sudo journalctl --since=today _SYSTEMD_UNIT=ssh.service

Maintain Journal

Vaccum only works on archived journal files
Send journals to remote host - systemd-journal-remote

# check disk usage
journalctl --disk-usage

# clean up & only leave 300M from archive
sudo journalctl --vacuum-size=300M

# before backup: sync
journalctl --sync

# view different journal files
journalctl --directory=xx

# merge journals
journalctl -m xx

Make journal entries manually

systemd-cat 📌

# create entry
echo "hello" | systemd-cat

# view
journalctl --no-pager | grep hello

3 - Logger

Manually create log entry

Example

# create log entry
logger shutting down

# result: in /var/log/messages
Jan 01 08:00:00 merikanto logger: shutting down

Options

-i	# record logger PID
-s	# output to stderr
-f	# log file content
-p	# specify log level
-u xx	# log directly to network socket xx

4 - Logrotate

Notes

Called on a regular basis via cron job
Also compress, delete, mail log file to user account
Logrotate status file - /var/lib/logrotate/status

Config - /etc/logrotate.conf

Compression
- Default - gzip
- Use xz - compresscmd xz
create
- Create new log file
- Options - file mode (0664), owner (root), group (kk)

# section in the config file 

# Rotate wtmp, which isn't handled by a specific program
/var/log/wtmp {
    monthly
    create 0664 root kk
    rotate 1
}

Directives	Explanation
`rotate n`	Number of old files to be maintained (e.g. `log.1`, `log.2`) If `n = 0`, rotated files are deleted
`weekly n`	0 - 6 (Sunday - Saturday). 7 means log is rotated every 7 days, regardless of the current day
`size n`	Rotate based on size (K, M, G)
`notifempty`	Don’t rotate, if log file is empty

5 - Mail

Linux follows the Unix method of handling email.

MTA - Mail Transfer Agent
- Send incoming mails to MDA / local user’s inbox
- Outbound messages to remote system: establish communication link with another MTA program
MDA - Mail Delivery Agent
- Deliver message to local user’s inbox
MUA - Mail User Agent
- Interface to display user meesages

Protocols

SMTP - Simple Mail transfer Protocol
- SMTP servers are known as MTA
Pull mail protocol
- POP - Post Office Protocol
- IMAP - Internet Message Access Protocol

Popular MTA packages

Sendmail
- Message forwarding
- User aliases
- Mail lists
Postfix - Simplicity
Exim - Sendmail replacement
qmail - Security as major design goal

Relays

At each step in a relay chain, email is altered
Each server adds a header to the email (can trace email back to its source)
Open relay - Relay mail from any computer to another 📌

Notes

Binary: /usr/bin/mail
Messages stored in /var/spool/mail
Operations
- Get log - /var/log/mail
- Read email - mail
- Check email queue - mailq / sendmail -bp
- Clear mail queue - sendmail -q
- Setup email alias (For redirction) - /etc/aliases ( Command: newaliases ) 📌

Maintain System Time

If messed up software clock, typically can make things right by rebooting.

1 - Linux Time

Two built-in clocks

Software clock - Linux use it when it’s running
Hardware clock - RTC (Real Time Clock), maintains time when computer is turned off

Gets power from system battery (CMOS battery)
x86-64 hardware maintains both hardware & software clocks, Linux provides tools to sync the two
Linux read the timestamp in UTC, then do the calculation, so time appears in local time

Set hardware clock from software clock - hwclock

# view hardware clock
hwclock --show

# view current rtc
hwclock -r

# sets rtc to use UTC
hwclock -u

System time services - If any is in active status, unable to set time with date

systemctl status ntpd - NTP
chronyd - improved ntpd
systemd-timesyncd

chronyd

Config file - /etc/chrony/chrony.conf
rtcsync - periodic update of RTC

# view system time sources
chronyc sources -v

# view time server stats
chronyc sourcestats

# view software clock performance
chronyc tracking

Set time with timedatectl instead

# turn off ntp
timedatectl set-ntp 0

# set time
timedatectl set-time "2022-04-07 08:00:00"

# turn on ntp
timedatectl set-ntp 1

2 - NTP

NTP - Network Time Protocol

Tiered hierarchy of time sources
- Top level (stratum-0 time servers):
  One or more hightly accurate time sources (atomic clocks / radio receivers)
Select NTP source - Select one with shortest network time delay
- Use ping 📌

Works by measuring packet’s round-trip time between server & client

Two systems exchange packets with embedded time stamps (mechanism to offset packet travel time)
Client adjust the time, so it’s in sync with timestamp from source (server)
Server improves system clock accuracy - /var/lib/ntp/ntp.drift

Config NTP

Package - ntp / ntpd
Config file - /etc/ntp.conf

# obtain server list & polling info
ntpq -p

# one-time clock setting on client
ntpd -g

# view software time sync  📌
ntpstat  

# set time manually
ntpdate

Leap-smearing

Google introduced the free public time server that uses NTP, and smear the leap second over course of time
Servers: time1.google.com ( from 1 to 4 )
NTP server list

Job Scheduling

Can also use systemd-run to schedule a job to run at a specific time.

1 - Cron

Crontab - The Cron Table

Running unsupervised - No user input
Two types of cron job
- System cron - different intervals in /etc/cron.xx (e.g. weekly, daily)
- User cron
Config file
- ```
**User** crontab - `/var/spool/cron/crontabs/` (Directory)
```
- System crontab (root) - /etc/crontab (File)
- /etc/cron.d

Edit cron job

1	02 4 * * * root run-parts /etc/cron.daily

Fields
- Minute - 0 - 59
- Hour - 0 - 23
- Day of the month - 1 - 31
- Month - 1 - 12
- Day of the week - 0 - 7 (Both 0 & 7 means Sunday ) 📌
Matching
- Comma-separated list (0, 6, 12) matches any specified values
- Specify range - 9-17 (9am to 5pm)
- Specify steps - */10 in minutes (every 10 min)

Use crontab

# create crontab from file
crontab -u merikanto [file]

-l	# show current crontab
-r	# remove current crontab
-e	# edit
-u xx	# specify user

2 - Anacron

Advantage over cron

More useful on systems that frequently shut down :
Ensures regular maintenance jobs are executed at reasonable intervals
Run from system startup script

Config anacron

Config file - /etc/anacrontab
Fields
- Frequency - in days
- Delay - in mins (keep system from overloaded at boot time)
- Identifier
- Command

1 2	# weekly job, delay for 10 min 7 10 cron.weekly run-parts /etc/cron.weekly

Create anacron jobs ( 2 ways )

Create cron job to run anacron 📌
Use startup script - Can slow down performance during boot time, if running time-consuming task

Disable any cron jobs that handled by anacron. Otherwise tasks will be performed twice.

3 - At

For running once-off jobs in the future

Time

Standard time - HH:MM
Standard date - DD.MM.YY
Keywords - noon, midnight, teatime (4pm)
Specific period - now + 2 hours

1 2	# use -f for filename at -f script.sh 08:00

Utilities

atd - the daemon, checks /var/soool/at 📌
atq - list pending jobs
atrm - remove job from queue (take job number, e.g. atrm 12)
batch - execute jobs when system load < 0.8