mbodo/lpi201.md

## lpi201.md

      
    Raw
  

              lpi201.md
            
          
    LPI-2 201 Notices

Topic 200: Capacity Planning


200.1 Measure and Troubleshoot Resource Usage - 6
200.2 Predict Future Resource Needs - 2

Topic 201: Linux Kernel


201.1 Kernel Components - 2
201.2 Compiling a kernel - 3
201.3 Kernel runtime management and troubleshooting - 4

Topic 202: System Startup


202.1 Customizing SysV-init system startup - 3
202.2 System Recovery - 4
202.3 Alternate Bootloaders - 2

Topic 203: Filesystem and Devices


203.1 Operating the Linux filesystem - 4
203.2 Maintaining a Linux filesystem - 3
203.3 Creating and configuring filesystem options - 2

Topic 204: Advanced Storage Device Administration


204.1 Configuring RAID - 3
204.2 Adjusting Storage Device Access - 2
204.3 Logical Volume Manager - 3## Topic 205: Network Configuration
205.1 Basic networking configuration - 3
205.2 Advanced Network Configuration and Troubleshooting - 4
205.3 Troubleshooting Network Issues - 4

Topic 206: System Maintenance


206.1 Make and install programs from source - 2
206.2 Backup operations - 3
206.3 Notify users on system-related issues - 1

 200.1 Measure and Troubleshoot Resource Usage


uptime

average load over the last 1, 5, and 15 min
1 cpu - 0.5 is 50 %, 1.5 is 150 %
2 cpu - 0.5 is 25 %, 1.5 is 75 %
y cpu - e.g 0.5x100/y


iostat

detailed CPU monitoring, I/O
%user   - utilization of user based processes
%nice   - utilization of alter process CPU priority
%system - utilization of kernel based process
%iowait - utilization of CPU waiting for disk I/O
%steal  - utilization of hypervisor's handle virtual CPU request
%idle   - utilization of CPU handling no requests
periodic reports
iostat -c 5 5

every 5 seconds 5 times


sar

like iostat only monitoring occurs overtime (10 min for example)


mpstat

to list all CPUs utilization
mpstat -P ALL

09:31:36 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle                      
09:31:36 PM  all    2.62    0.02    1.47    0.25    0.00    0.00    0.00    0.00    0.00   95.64                      
09:31:36 PM    0    3.52    0.03    1.73    0.07    0.00    0.00    0.00    0.00    0.00   94.64                      


memory monitoring

must have both RAM + swap space


free

used - amount of memory used by tmpfs
-b bytes, -m mega, -g giga, -h human readable (so b,m,g according to values)


vmstat

detail memory monitoring


iostat

display statistic about disks I/O
iostat -d


sar -d same statistic as iostat, data by default in 10-min intervals


lsof

COMMAND  - cmd that opened the file
PID      - PID of process
USER     - user who started cmd
FD       - file descriptor, e.g txt program test (code and data)
TYPE     - type of node, e.g DIR directory, REG regular node, IPv4 socked node
DEVICE   - device number, e.g SATA drive, pseudo device
SIZE/OFF - file size or file offset
NODE     - node value, based on node type
NAME     - absolute pathname to the node
Examples:
List pids of files using a characters

$ lsof -t -c tmux
6336
7893

List commands with characters tmux
lsof -c tmux

List nodes of processes runnning on network port
lsof - i TCP:22

List for specific PID
lsof -p 100

List all file within a directory
lsof +d /usr/bin


netstat
Display routing table
netstat -r

Display statistic
netstat -s

Display -p PID, -l sockets in listening state, -n don't resolve IP addreses, -t TCP only
netstat -plnt


ps
pstree
top

default interval is every 2s


w

Links:

[use-vmstat-to-monitor-system-performance] (https://www.linode.com/docs/uptime/monitoring/use-vmstat-to-monitor-system-performance/)
[linuxatemyram] (https://www.linuxatemyram.com/)
[pagefaults and memory swap] (https://scoutapm.com/blog/understanding-page-faults-and-memory-swap-in-outs-when-should-you-worry)
[Anatomy of Linux dynamic libraries] (https://developer.ibm.com/tutorials/l-dynamic-libraries/)

 201.1 Kernel Components


Kernel Components

core, kernel modules


Kernel Documentation

/usr/src/linux/Documentation (kernels < 3.0)
/usr/share/doc/kernel-doc*/Documentation (kernels < 3.0)
https://www.kernel.org/doc


Kernel Tweaking

/proc/sys
e.g change file-max temporarily
cat /proc/sys/fs/file-max
3185849

echo 60000 > /proc/sys/fs/file-max

cat /proc/sys/fs/file-max
60000


Kernel Images

zImage, Intel-based, compresed kernel image for low memory
bzImage, big-z-Image, greater than 512 kb  for high memory
naming is vmlinuz-version.arch
to find out if we have zImage or bzImage
$ file /boot/vmlinuz-3.XX.X-XXX.XX.X.XXX.x86_64
Linux kernel x86 boot executable bzImage, version ....


 201.2 Compiling a kernel


Compiling Kernel

https://www.kernel.org/
unpackt to /usr/src
cd /usr/src/kernel-version
make clean
if additional error occur later in process use make mrproper, make distclean
create configuration file

make config, configure each parameter
make menuconfig, tui to choose kernel parameters
modules kernel parameters:

N, exclude
M, avialable as loadable kernel module
Y, include in kernel image


result of configuration saved to /usr/src/linux-version/.config
make xconfig, gui based on QT to choose kernel parameters
make gconfig, gui based on GTK+ to choose kernel parameters


build kernel make bzImage, Note: make zImage will fail too large
image compiled to ./arch/x86_64/boot/bzImage
build kernel modules make modules
make modules_install, to copy to /lib/modules/kernel-version.x86_64
copy cp /usr/src/kernel-version/arch/x86_64/boot/bzImage /boot/vmlinuz-version.x86_64
create mkinitrd /boot/initramfs-version.x86_64.img version, call dracut utility
edit /boot/grub/grub.conf


Additional Make targets

make all, build kernel and modules together
make oldconfig, backup of /usr/src/linux-version/.config only for make config target, other backup automatically e.g make menuconfig
make rpm-pkg - kernel RHEL RPMs src/bin
make binrpm-pkg - kernel RHEL RPMs bin
make deb-pkg - kernel Debian package


 201.3 Kernel runtime management and troubleshooting


Kernel modules

stored in /lib/modules, hierarchicaly by categories /lib/modules/fs
if module is copy manually it critical to execute depmod command
depmod probes all modules under /lib/modules/kernel_version/kernel
depmod generate:

/lib/modules/kernel-version/modules.dep - contains list off all kernel module dependencies
/lib/modules/kernel-version/*map files


depmod generate dependencies for current kernel, to execute for different depmod version e.g depmod 2.6.38.68
list loaded kernel modules
$ lsmod | head 

Module                  Size  Used by
vhost_net              22499  2 

Used by 2 - how many 'clients' uses that module, e.g processes, other modules, kernel itself
...


loading modules into memory

automatically
manually:

insmod

e.g fat module

$ insmod /lib/modules/kernel-version/fs/fat/vfat.ko

in case of unsatisfied dependencies, insmod doesn't handle dependencies

$ grep -i --color 'vfat' /lib/modules/3.10.0-957.21.2.el7.x86_64/modules.dep
kernel/fs/fat/vfat.ko.xz: kernel/fs/fat/fat.ko.xz

$ insmod /lib/modules/kernel-version/fs/fat/fat.ko
$ insmod /lib/modules/kernel-version/fs/fat/vfat.ko

modprobe - handle module dependencies, read /lib/modules/kernel-version/modules.dep

$ modprobe vfat


unload modules from memory
rmmod fat

modprobe -r fat


listing modules informations
$ modinfo cdrom
param:       lockdoor


set temporary

$ modprobe cdrom lockdoor=0


/proc/sys filesystem

/proc/sys im memory filesystem related to kernel
equivalent to modprobe cdrom lockdoor=0 is echo "0" > /proc/sys/dev/cdrom/lock, this will change the value of module without reloading it


sysctl

get parameter
$ sysctl -a | grep dev.cdrom.lock
dev.cdrom.lock = 1


set parameter, not permanent
sysctl dev.cdrom.lock=0


in case of invalid value
$ sysctl dev.cdrom.lock="abc"
error: "Invalid argument" setting key "dev.cdrom.lock"


permanent changes in /etc/sysctl.conf
some linux distributions has /etc/sysctl.d


Module aliases

/etc/modprobe.d - directory to adjust how nodules are loaded
format: install, remove, alias


lspci

detailed
$ lspci -v


with kernel modules
$ lspci -k


lsdev

RHEL base distribution
get infos from /proc/dma,/proc/ioports,/proc/interrupts


lsusb

detailed informations
$ lsusb -v -d 1d6b:0002


udev system

udevd daemon
udev rules /etc/udev/rules.d
udevadm monitor


Links:

[compiling-linux-kernel] (https://www.cyberciti.biz/tips/compiling-linux-kernel-26.html)

 202.1 Customizing SysV-init system startup


runlevels
   LSB           RHEL          Debian
0  Halt          Halt          Halt
1  Single        Single        Single
2  Mul. no GUI   Mul. no GUI   Mul. with GUI
   no Net        no NFS
3  Mul. no GUI   Mul. no GUI   Mul. with GUI
4  Not defined   Copy of 3     Mul. with GUI
5  Mul. with GUI Mul. with GUI Mul. with GUI
6  Reboot        Reboot        Reboot


non standard not used 7,8,9 levels
some dist. has S wicht stands for 1 level


/etc/init files

init process read /etc/inittab file with format
id:runlevel:keyword:command 
1 :2345    :respawn:/sbin/mingetty tty1

Keywords:
1, initdefault - default runlevel
2, sysinit     - when system boots, regardles of which level system booted to, executed first before other commands
3, wait        - the init process will wait for commands to complete before continuing
4, ctraltdel   - "trap", keyboard combination
5, powerfail   - signal from UPS is it run out of power
6, powerokwait - powerofwait sent to init process
7, respawn     - if user log off the process needs to be restarted


boot scripts

/etc/rc.d/rc.sysinit - base state, essentially same as single user mode
/etc/rc.d/rc         - runlevels 1 - 6
l5:5:wait:/etc/rc.d/rc 5

Boot process:
1, finds /etc/rc5.d and begin with K letter execute with stop 
2, finds /etc/rc5.d and begin with S letter execute with start


start scripts
Example: 

/etc/rc5.d/S55sshd the file is actually a link to /etc/init.d

/etc/rc5.d/S55sshd -> /etc/init.d

Notes:

- stopping services in /etc/rc5.d while levels could be changed manually
- execute order is from higher to lower, S11 -> S66
- all scripts starts under root user 


/etc/init.d/

user can manually start/stop/restart services e.g /etc/init/sshd restart


modify runlevels

Debian update-rc.d
1. create /etc/init.d/myserver
2. chmod -v 0544 /etc/init.d/myserver
3. update-rc.d myserver defaults

The update-rc.d will create links with K script links from /etc/init.d in /etc/rc0.d, /etc/rc1.d, /etc/rc6.d
and S script links form /etc/init.d in /etc/rc2.d, /etc/rc3.d, /etc/rc4.d, /etc/rc5.d


RHEL chkconfig
1. create /etc/init.d/myserver
2. chmod -v 0544 /etc/init.d/myserver
3. chkconfig

A line in script muss exists # chkconfig 2345 55 25 , where 2345 are runlevels, S55myserver, K25myserver then use

chkconfig --add myserver


update-rc.d
Add to multiuser:

update-rc.d myserver multiuser

Add to levels explicitly:

update-rc.d myserver start 40 3 4 5. stop 15 0 1 2 6 .

= Notes:

value with runlevels must end with "."

start 40 3 4 5.
S40 /etc/rc3.d /etc/rc4.d /etc/rc5.d

stop 15 0 1 2 6 .
K15 /etc/rc0.d /etc/rc2.d /etc6.d

=

Remove service:

update-rc.d myserver remove


chkconfig
List services:

chkconfig --list

Turn off service

chkconfig bluez off

Turn on service:

chkconfig  bluez on

Turnoff for runlevel 2:

chkconfig bluez --level 2 off

Turnon for runlevel 2:

chkconfig bluez --level 2 on


switch runlevels

list current
$ runlevel
N 5


switch runlevel
Directly to runlevel 3:

init 3

To runlevel 3 with delay(seconds):

init -t 60 3


 202.2 System Recovery


boot process

BIOS/UEFI Phase

UEFI - larger boot partitions, GUID partition tables


bootloader phase

LILO, Grub2, legacy Grub
stored in MBR, first 215 bytes of the boot disk
install GRUB on first SATA hard drive
grub-install /dev/sda


kernel phase

bootloader loads the kernel and passes control to the kernel
kernel than configure itself and loads kernel modules
1, configure system to start assigning memory addresses to software
2, probes/configure hw
3, uncompress initrd/initramfs image, contains kernel modules
4, initialize meta devices LVM, RAID devices
5, mounts /root filesystem read-only
6, starts post-kernel phase


post kernel phase

once kernel complete all its tasks, it start a process thats boot OS
systemd, upstart, SysV-init


GRUB

Differencies between legacy GRUB and GRUB 2

GRUB2 supp. more OS
GRUB2 may use UUID
GRUB2 supp. RAID/LVM
GRUB2 multiple conf files, legacy GRUB 1 conf file


Legacy GRUB configuration

/boot/grub/grub.conf
Configuration settings:

default     - which title to boot, value numeric, how title appear in /boot/grub/grub.conf file
timeout     - wait to user choose
splashimage - 14 colors, 640 x 480, .xpm.gz
hiddenmenu  - menu not displayed, only coutdowntimer, if key is pressed countdown ends, menu is displayed

Title:
title CentOS (3.10.0-404.el6)
      root (hd0,0)       
      kernel /vmlinuz-2.6.18-406.el5 ro root=/dev/VolGroup00/LogVol00 rhgbquiet
      initrd /initrd-2.6.18-406.el5.img

title  - display in menu during a boot
root   - device contain boot files, kernel and initrd, e.g (0=/dev/[hs]da, 1=/dev/[hs]db)
kernel - location of kernel to by loaded
initrd - localtion of the initrd file, relative to the device in root parameter


modify legacy GRUB dirung boot
1, select alternative title to boot
2, modify parametres for a title       (title -> e key)
3, GRUB commands via GRUB command line (c key)


booting legacy GRUB to single user mode
Recovery root password

press a key -> add "s" to end of kernel parameters -> Enter -> press b -> boot to single user mode -> use "passwd"


securing legacy grub
$ grub-md5-crypt
Password
Retype password:
$1%YT

add the line to /boot/grub/grub.conf

...
splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenu
password --md5     $1%YT
title CentOS (3.10.0-404.el6)
       root (hd0,0)       
...

to edit the GRUB legacy configuration press 'p' -> password


GRUB2 configuration

/boot/grub2/grub.cfg RHEL-based
/boot/grub/grub.cfg  Debian-based
never edit /boot/grub2/grub.cfg or /boot/grub/grub.cfg directly, use /etc/default/grub, /etc/grub.d
/etc/grub/default, global GRUB2 options
GRUB_DEFAULT - default number to boot
GRUB_SAVEDEFAULT remember last used and uses as default for nex boot
GRUB_TIMEOUT sec. to wait before choose auto. default, -1 infinitely
GRUB_TIMEOUT_STYLE if hidden, menu and title not visible only countdown
GRUB_CMDLINE_LINUX - global parameters for every kernel title
GRUB_CMDLINE_LINUX_DEFAULT two titles regular/recovery, this is params for regular title


/etc/grub.d, specific changes
save GRUB2 changes
RHEL-based
$grub2-mkconfig

Debian-based
$update-grub


GRUB2 Titles
grub2-mkconfig, update-grub generate menuentry settings to /boot/grub2/grub.cfg, /boot/grub/grub.cfg

then
GRUB_DEFAULT='CentOS (3.10.0-404.el6) 6 (Core)'

to create custom title use /etc/grub.d/40_custom and run grub2-mkconfig, update-grub


booting to singel user mode GRUB2
Password recovery:

1, press e key -> line begind with linux/linux16 add init=/bin/sh to end of line -> boot wit Ctrl + x
2, mount -o remount,rw /
3, if selinux enabled touch /.autorelabel
4, passwd
5, exec /sbin/reboot


securing GRUB2
1, create one superuser - has access to all features of GRUB2 in /etc/grub.d/01_users
   cat <<EOF
   set superusers="bo"
   password bo bospassword
   EOF
2, add /etc/grub.d/40_custom
   menuentry 'test3' {
   set root=(hd0,3)
   linux   /vmlinuz
   }
3, if don't want to use plain text passwords generate pasword hash 

   $grub2-mkpasswd-pbkdf2
   
4, edit /etc/grub.d/01_users
  cat <<EOF
  set superusers="bo"
  password_ pbkdf2 bo
  grub.pbkdf2.sha512.10000.560799R9ED..
  EOF   


Fixing file systems


fixing only properly unmounted filesystems
fsck /dev/sdb1
fsck -y /dev/sbd1

 203.1 Operating the Linux filesystem


fundalmentals

virtual filesystem e.g linux /
physical, e.g partition, LVM
automatically mounted /etc/fstab


filesystem types

ext2, ext3, ext4, btrfs, xfs, UDF, ISO9660, HFS


umount

/dev/sda1 -> /boot, umount /dev/sda1 or umount /boot


mount

display mounted filesystems
mount

displayed from /etc/mtab and managed by mount/umount commands


same as /etc/mtab is /proc/mounts managed by kernel


mounting fs manually with options
$ mount -o usrquota,grpquota /dev/sda1 /boot
$ mount | grep /dev/sda1
/boot type ext4 (rw,usrquota,grpquota)


mounting fs manually with filesystem type
$ mount -t ext4 /dev/sda1 /boot
$ mount | grep /dev/sda1
/boot type ext4 (rw)


troubleshooting unmount

lsof /boot/*
fuser /boot
/boot:   1234c

c current directory
e executable
f open file
F open file for writing
r root directory
m mmap'ed file or shared memory


mounting files automatically

/etc/fstab
format:
device_to_mount | mount_point | fs_type | mount_options | dump_level | fsck_value

/dev/sdb1       | /data       | etx4    | default       | 0          | 1 


device descriptors

/dev/sdb1
e2label
Create:

e2label /dev/sdb1 data

in /etc/fstab

LABEL="data" /data ext4 defaults 0 1

Display:

$ e2label /dev/sdb1
data

$ blkid | grep sdb1
/dev/sdb1 ... LABEL="data"

Note: May cause conflict when disk is switched between physical machines


UUID
Get UUID:

$ blkid | grep sdb1
/dev/sdb1 UUID=374e1708-eeec

in /etc/fstab

UUID=374e1708-eeec /data ext4  defaults 0 1


LV (logical volumes), doesn't change the name


mount options

default
rw,suid,dev,exec,auto,nouser,async,relatime


sync, force all data stored from memory to the hardisc


swap space

swapon -s
List all active swap devices

Filename                    Type          Size          Used          Priority
/dev/dm-1                 partition     1048731         0             -1

Priority - tell which swap device should use as first


permanentlly in /etc/fstab
remove swap device
$ swapoff /dev/dm-1


 203.2 Maintaining a Linux filesystem


ext2

1993
only one timestamp
today only for removable devices


ext3

journal
max. 2TB one file
max 16TB overall


ext4

handle large files
max 16TB one file
max 1EB overall


xfs

B+tree structure
max 8EB one file
max 16XB overall


btrfs

good at dealing with large chunks of read and writes
B-tree structure
snapshots


Creating filesystems

partition create with fdisk, gdisk
mkfs, front-end command to command designed for specific filesystems e.g mkfs.ext4
$ mkfs -t ext4 /dev/sdb1


specifics options for mkfs.* are passed to through mkfs
$ mkfs -t ext4 -m 10 /dev/sdb1

-m is specifics option


ext-based filesystem tools

fsck, frontend utility for specifics fsck.*
dumpe2fs, display details about ext* filesystem
Displays:

Filesystem features, Default mount options, Inode count, Reserved block count, Block size


tune2fs, not every filesystem features could be changed
$ tune2fs -m 0 /dev/sdb1


debugfs, debugging operations, filesystems not currently mounted

possible to undelete file
$ umount /data
$ debugfs /dev/sdb1
$ lsdel
$ undel file


xfs based filesystem tools

xfsdump, backup xfs filesystem
$ xfsdump -f /tmp/boot_back /boot

Ask for:
1, label       - name of backup 
2, media label - name for device

By default a full backup is made.

Use -l to specify the backup level


xfsrestore, restore files from backup
Restore to current directory

xfrestore -f /backup/location .

To list files in backup

xfsrestore -t -f /tmp/boot_back | grep -v "^xfsrestore" | head

Restore only one file interactive

$ xfsrestore -i -f /tmp/boot_back /tmp
$ ls
$ add file_name
$ extract


xfs_info, basic filesystem informations
$ xfs_info /dev/sdb1


xfs_repair, similar to fsck
xfs_check
Equal to 

xfs_repair -n 

-n no repair


xfs_repair,xfs_check filesystem must be unmounted


creating swap devices

swap partition
Setup with temprary on

$ mkswap /dev/sdb1
$ swapon /dev/sdb1

Permanent

/etc/fstab

UID=03dt71dc-c278 swap               defaults       0 0


swap file
Setup

$ dd if=/dev/zero of=/var/new_swap bs=1M count=200
$ mkswap /var/new_swap
$ swapon /var/new_swap


monitor SMART devices

SMART daemon smartd
When smartd is running
Overall heath

smartctl -H /dev/sdb

Detailed information

smartctl -i /dev/sdb


Links:

[Linux namespaces] (https://lwn.net/Articles/531114/#series_index)

 204.3 Logical Volume Manager


LVM essentials

convert to physical volumes
Have 3 drives

pvcreate /dev/sdb /dev/sdc /dev/sdd

Physical volume "/dev/sdb" successfully created


create volume group
vgcreate VG0 /dev/sdb /dev/sdc /dev/sdd


physical extents, default 4MB
To change when create VG (Volume Group)

vgcreate -s 16MB VG0 /dev/sdb /dev/sdc /dev/sdd

if extent 4MB -> LV could by build e.g 20 MB from 5 extents -> limit in LVM1 is 65 534 so for (4MB only 250GB for 1 LV), LVM2 doesn'thave it


display VG information
Default info display:

vgdisplay VG0

Info about any PV, LV:

vgdisplay -v


display info about PV
pvdisplay /dev/sdb


create logical volumes
lvcreate -l 6 -n lv0 VGO

-l, extens

lvcreate -L 120MB -n lv0 VG0

-L MB

lvcreate -i 3 -l 6 -n lv1 VG0


device naming
LVM1:

/dev/VG0
/dev/VG0/lv0 -> ../dm-3
/dev/VG0/lv1 -> ../dm-4

LVM2:
/dev/mapper/VG0-lv0 -> ../dm-3
/dev/mapper/VG0-lv1 -> ../dm-4


 205.1 Basic networking configuration


ifconfig

change network setting temporarily
ifconfig eth0 192.168.1.16 netmask 255.255.255.0 broadcast 192.168.2.255


enable promiscuous mode
ifconfig eth0 promisc


disable promiscuous mode
ifconfig eth0 -promisc


turn off ARP (IP-to-MAC)
ifconfig eth0 -arp


turn on ARP (IP-to-MAC)
ifconfig eth0 arp


arp

display arp table
$ apr

Address              HWtype  HWaddress           Flags Mask            Iface
192.168.X.XX         ether   30:XX:XX:XX:XX:XX   C                     eth0


delete arp entry, in case of HW card replacement
arp -i eth0 -d 192.168.X.XX


route

display routing table
route -n

Kernel IP routing table
Destination     Gateway     Genmask       Flags Metric Ref     Use Iface
0.0.0.0         10.23.2.1   0.0.0.0       UG    0      0         0 eth0
10.23.2.0       0.0.0.0     255.255.255.0 U     1      0         0 eth0

UG (UP Gateway)
U  (UP)

To reach 10.23.2.132 the longest destination prefix principle is used, CIDR:
- 0.0.0.0   matches 0
- 10.23.2.0 matches 24 longest is used


add local subnet routing
route add -net 10.23.3.0 netmask 255.255.255.0 dev eth0


delete gateway
route del -net 10.23.3.0 netmask 255.255.255.0 gw 10.23.3.1


delete default gateway
route del default


add default gateway
route add default gw 10.23.2.1


ip

include all the functionality of ifconfig, arp and route
Example of equivalent commads
ifconfig                               | ip addr show, ip link show   
ifconfig eth0 192.168.1.16 netmask 
255.255.255.0 broadcast 192.168.2.255  | ip addr add 192.168.1.16/24 broadcast 192.168.2.255 dev eth0
ifconfig eth0 promisc                  | ip link set eth0 promisc on
ifconfig eth0 -promisc                 | ip link set eth0 promisc off
ifconfig eth0 -arp                     | ip link set eth0 arp off
ifconfig eth0 arp                      | ip link set eth0 arp on
arp                                    | ip neigh show
arp -i eth0 -d 192.168.1.11            | ip neigh del 192.168.1.11 dev eth0
route                                  | ip route show
route add -net 192.168.2.0 netmask
255.255.255.0 dev eth0                 | ip route add 192.168.2.0/24 dev eth0
route del -net 192.168.1.11 netmask
255.255.2555.0 dev eth0                | ip route del 192.168.1.11/24 dev eth0
route del default                      | ip route del default
route add default gw 192.168.1.1       | ip route add default via 192.168.1.1 


  205.2 Advanced Network Configuration and Troubleshooting


ping

ICMP, Internet Control Message Protocol
ping with limit
ping -c 3 192.168.1.11


ping with flood
ping -f -c 5000 192.168.1.11


nmap

probe which network ports are open
default scan
$ nmap 192.168.1.11

PORT   STATE SERVICE
22/tcp open  ssh


scan UDP ports
$ nmap 192.168.1.11


scan UDP specifics port range with version
$ nmap -sUV -p U:1-10 192.168.1.11

PORT   STATE  SERVICE     VERSION
1/udp  closed tcpmux
2/udp  closed compressnet
...
9/udp  closed discard
10/udp closed unknown


scan TCP specifics port range
$ nmap -p 1-65535 192.168.1.11

Not shown: 65533 closed ports
PORT     STATE SERVICE
5601/tcp open  unknown


scan to determine operating system type
$ nmap -O 192.168.1.11


scan entire network
$ nmap -sP 192.168.1.0/24


scan list of network interfaces
$ nmap --iflist


netstat

display statistic
$ netstat -s


display routing table
$ netstat -r


display network statistic for each interface
$ netstat -i


lsof

list open network sockets
$ lsof -i


list open network sockets for 22 port
$ lsof -i:22


nc

listen for connection on port, will end after client disconnect
$ nc -l 3333


listen for connection on port, will remain after client disconnect
$ nc -lk 3333


closes a connection with connection timeout
for 30s

$ nc -w 30 server_hostname_ip 3333


UDP connection
Server: 

nc -l -u 3333

Client:

echo "Hello world" | nc -u server 3333


Display open ports
$ nc -z localhost 1000-4000


tcpdump

packet sniffer
limit captured packets for IF
tcpdump -c 5 -i eth0


capture packets for IF and port
tcpdump -i eth0 port 3333


capture packets for IF and source only
tcpdump -i eth0 src client_ip


capture packets for IF and destination only
tcpdump -i eth0 dst client_ip


capture packets for IF and source only to file
tcpdump -i eth0 src client_ip -w tcpdump.out


read packets for IF and source only to file
tcpdump -r tcpdump.out


 205.3 Troubleshooting Network Issues


/etc/resolv.conf

client-side DNS settings
nameserver 192.168.1.1


max up tu 3 DNS server configurations
search setting, 6 search domain max, each search line 256 char max
nameserver 192.168.1.1
search onedomain.com

then

ping -c 1 www

PING www.onedomain.com (192.168.1.7) ...


timeout response fo query DNS server
nameserver 192.168.1.1
search onedomain.com
options timeout: 3


number of attempts
nameserver 192.168.1.1
search onedomain.com
options timeout: 3
options attempts: 2


round robin distribution
nameserver 192.168.1.1
search onedomain.com
options timeout: 3
options attempts: 2
options route


3 ways to populate

Hand modified
DHCP
NetworkManager


hostname

/sbin/hostname
non-RHEL: /etc/hostname
RHEL: /etc/hostname, /etc/sysconfig/network
Debian,SUSE: /etc/hostname, /etc/HOSTNAME


/etc/network/interfaces

Debian-based primary network interface conf. file
dhcp IPv4 configuration
auto eth0
iface eth0 inet dhcp


dhcp IPv6 configuration
auto eth0
iface eth0 inet6 dhcp


assign static
iface eth0 inet static
address 192.168.1.7
netmask 255.255.255.0
gateway 192.168.1.1


second network interface
iface eth1 inet static
address 192.168.1.7
netmask 255.255.255.0
gateway 192.168.1.1


/etc/sysconfig/network-scripts

RHEL-based primary network interface conf. directory
/etc/sysconfig/network-scripts/ifcfg-eth0, interface eth0 conf. file
dhcp client configuration
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=dhcp


dhcp IPv6 client configuration
/etc/sysconfig/network
NETWORKING_IPV6=yes

/etc/sysconfig/network-scripts/ifcfg-eth0
DHCPV6C=yes     


assign static
DEVICE=eth0
IPADDR=192.168.1.7
NETMASK=255.255.255.0
BROADCAST=192.168.1.254
ONBOOT=yes
BOOTPROTO=none


Log files and Tools

/var/log/syslog, Debian-based primary log file
/var/log/messages, RHEL-based primary log file
/var/log/dmesg, kernel ring buffer


traceroute

gateway hops, ICMP packets


/etc/hosts.allow, /etc/hosts.deny

TCP wrappers
server programs compiled with libwrap library
Secure Shell Server Daemon (sshd)
ldd /usr/sbin/sshd | grep -i --color libwrap
sshd determine which connection is allowed/denied using a /etc/hosts.allow,/etc/hosts.deny
fundamentals: /etc/hosts.allow -> no match -> /etc/hosts.deny -> match then deny
rules:
service_list: client_list [options]

client_list:
- IP address
- Network
- Entire domain
- ALL
- Local
- Unknown
- Known

e.g

---
/etc/hosts.deny
sshd: onedomain.com

all clients are permitted except form the onedomain.com
---

---
/etc/hosts.deny
xined,sshd: onedomain.com

all clients are permitted except form the onedomain.com for multiple services xined, sshd
---

---
/etc/hosts.deny
ALL: onedomain.com

all clients are permitted except form the onedomain.com for ALL services
---

---
/etc/hosts.allow
ALL: test.onedomain.com
/etc/hosts.deny
ALL: .onedomain.com

only clients from test.onedomain.com has ALL services permitted, all clients from entire domain .onedomain.com for ALL services are denied
---


NetworkManager

disable network manager SysV
chkconfig NetworkManager off


check in systemd
systemctl status NetworkManager


device query info
nm-tool


GUI based tool
nm-connection-editor


Links:

[unable-to-add-a-static-route-sioaddrt-network-is-unreachable] (https://serverfault.com/questions/581159/unable-to-add-a-static-route-sioaddrt-network-is-unreachable)
[what-is-the-difference-between-virbr-and-vnet] (https://unix.stackexchange.com/questions/52855/what-is-the-difference-between-virbr-and-vnet)
[route-table-explanation] (https://askubuntu.com/questions/72733/route-table-explanation)
(socat) [https://medium.com/@copyconstruct/socat-29453e9fc8a6]
(socat source) [https://github.com/craSH/socat]
[Broadcast_address] (https://en.wikipedia.org/wiki/Broadcast_address)
[Bitwise_operation] (https://en.wikipedia.org/wiki/Bitwise_operation)
[Ancient_Egyptian_multiplication - bitwise multiplication of 2 numbers] (https://en.wikipedia.org/wiki/Ancient_Egyptian_multiplication)