imShakil/lpi101.md

## lpi101.md

      
    Raw
  

              lpi101.md
            
          
    LPI-1 101 Notices

Topic 101: System Architecture


101.1 Determine and configure hardware settings
101.2 Boot the system
101.3 Change runlevels / boot targets and shutdown or reboot system

Topic 102: Linux installation and package management


102.3 Manage shared libraries
102.4 Use Debian package management
102.5 Use RPM and YUM package management

Topic 103: GNU and UNIX commands


103.1 Work on the command line
103.3 Perform basic file and directory management
103.4 Use streams, pipes, and redirects
103.5 Create, monitor, and kill processes
103.6 Modify process execution priorities

Topic 104: Devices, Linux Filesystems, Filesystem Hierarchy Standard


104.1 Create partitions and filesystems
104.3 Control mounting and unmounting of filesystems
104.5 Manage file permissions and ownership
104.6 Create and change hard and symbolic links

 101.1 Determine and configure hardware settings


IRQ - Interrupt request

  cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:        118          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  1:          0          0          0         11          0          0          0          0  IR-IO-APIC   1-edge      i8042
  8:          0          1          0          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0        253          0          0          0          0          0          0  IR-IO-APIC   9-fasteoi   acpi
 12:          0          0        149          0          0          0          0          0  IR-IO-APIC  12-edge      i8042
 16:          0          0          0          0          0         29          0          0  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
 17:          0          0          0          0          0          0        959          0  IR-IO-APIC  17-fasteoi   snd_hda_intel:card1
 18:          0          0          0          0          0          0          0          0  IR-IO-APIC  18-fasteoi   i801_smbus
 23:          0          0          0          0          0          0         35          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0    9705150          0          0          0  IR-PCI-MSI 327680-edge      xhci_hcd
 26:          0          0          0          0          0          0          0     262407  IR-PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 27:          0          0          0         11          0          0          0          0  IR-PCI-MSI 360448-edge      mei_me
 28:          0          0          0          0    1826232          0          0          0  IR-PCI-MSI 31457280-edge      enp60s0
 29:          0          0          0          0          0       1074          0          0  IR-PCI-MSI 442368-edge      snd_hda_intel:card0
 30:          0          0          0          0          0          0          0       3990  IR-PCI-MSI 30932992-edge      iwlwifi
 31:    2178755          0          0          0          0          0          0          0  IR-PCI-MSI 524288-edge      nvidia
...

Legend/Example:

0-31 - IRQ
CPU0-7 - number of interrupts per IRQ
IR-PCI-MSI 31457280-edge - interrupt type
enp60s0 - device / kernel module


I/O Addresses

cat /proc/ioports

0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard


DMA Addresses

cat /proc/dma

4: cascade


/sys - sysfs virtual file system

files represent object attributes
directories represent object themselves
top-level directories represent major subsystems
symbolic links represent relationships between objects

mount sysfs

mount -t sysfs sysfs /sys


HAL - Hardware abstraction level (replaced by udev)

Links:


Wiki HAL


D-Bus - Desktop Bus

Links:


Wiki D-Bus


udev - responsible for the dynamic device management needed for hot plugging devices

information about configured and active devices is contained in the /dev

b - block device
c - character device
d - directory
l - symbolic link to (dev,proc,run)


create device files
mknod /dev/sda1 b 8 2

b - block device
8 - major number
2 - minor number


daemon systemd-udevd.service, manage udev events
default udev rules /usr/lib/udev/rules.d
local udev rules (may override defaults) /etc/udev/rules.d
udev configure /etc/udev/udev.config
udevadm program is an administration tool for udevd e.g.
udevadm info --query=all –-name=/dev/sda

P: /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NSAG521465L
E: DEVLINKS=/dev/disk/by-id/wwn-0x50025388a0a87a4c /dev/disk/by-path/pci-0000:00:1f.2-ata-1 /dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NSAG521465L

P - sysfs device path
N - device node (in the /dev)
S - symbolic link, placed by udevd in /dev
E - additional device info from udevd rules


monitor udevadm events
Monitor messages, incoming from kernel, udevd sends out to other programs

udevadm monitor 

Only kernel and match scsi only

udevadm monitor --kernel --subsystem-match=scsi


Links:


Wiki udev


lpci - list all PCI devices

looks up to /usr/share/hwdata/pci.ids
new device, query PCI ID server database using DNS lookup

Query and write to ~/pciids-cache

lspci -q 

Detailed pci list

lspci -vvv

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

Show names and numbers

lspci -nn

...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204M [GeForce GTX 965M] [10de:13d9] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
...

Tree of devices in slot 00

lspci -tvv -s \:00

-+-[0000:3c]---00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
 +-[0000:3b]---00.0  Intel Corporation Wireless 7265
 +-[0000:01]-+-00.0  NVIDIA Corporation GM204M [GeForce GTX 965M]
 |           \-00.1  NVIDIA Corporation GM204 High Definition Audio Controller
 \-[0000:00]---00.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller

List by vedor id (in this example Nvidia 10de)

lspci -d 10de\:

01:00.0 VGA compatible controller: NVIDIA Corporation GM204M [GeForce GTX 965M] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)


lsusb - list all USB buses/devices

support USB 1,2,3
USB 1.1 (12Mbps),  driver OHCI(UHCI)
USB 2.0 (480Mbps), driver EHCI
USB 3.0 (5Gbps),   driver XHCI

lsusb

Bus 002 Device 002: ID 8087:8000 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp.
...

Tree view

lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/14p, 480M
  |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
  |__ Port 2: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
...

Device ID view

lsusb -d 0951\:

Bus 003 Device 004: ID 0951:1660 Kingston Technology Data Traveller 108


Kernel modules

lsmod - basically it formats /proc/modules

Module                  Size  Used by
usbcore               286720  9 uvcvideo,usbhid,snd_usb_audio,ehci_hcd,xhci_pci,snd_usbmidi_lib,btusb,xhci_hcd,ehci_pci
usb_common             16384  1 usbcore
i8042                  32768  1 asus_nb_wmi
serio                  28672  9 serio_raw,atkbd,psmouse,i804

Used by: 
usb_common 16384  1 usbcore

[usb_common] - actual module
[1] - number of process, other modules which are using the kernel module (in this case usb_common module)
[usbcore] - relies on usb_common


modinfo - show info about kernel module

modinfo serio

filename:       /lib/modules/4.15.6-1-ARCH/kernel/drivers/input/serio/serio.ko.xz
license:        GPL
description:    Serio abstraction core
author:         Vojtech Pavlik <vojtech@ucw.cz>
srcversion:     91E703BAD5EBF435706CF36
depends:
retpoline:      Y
intree:         Y
name:           serio
vermagic:       4.15.6-1-ARCH SMP preempt mod_unload modversions

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204M [GeForce GTX 965M] [10de:13d9] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

Usefull for scripts:

modinfo -F vermagic serio
4.15.6-1-ARCH SMP preempt mod_unload modversions

modinfo -F filename serio
/lib/modules/4.15.6-1-ARCH/kernel/drivers/input/serio/serio.ko.xz


insmod - loads module only, not dependencies (it expects they are already loaded)

insmod /lib/modules/4.15.6-1-ARCH/kernel/net/ipv4/netfilter/ip_tables.ko.xz


modprobe - Add or remove kernel modules, automatically loads any depended-on module

Load the ip_tables module and dependecies:

modprobe ip_tables

Dry run with verbose mode, perform checks except module insertions:

modprobe -v --dry-run ip_tables

modprobe --show-depends nfsd
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/net/sunrpc/sunrpc.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfs_common/grace.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/lockd/lockd.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfs_common/nfs_acl.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfsd/nfsd.ko.xz 

Remove module:

modprobe -r ip_tables


rmmod - remove kernel module, operates on single module, if depend on other module or in use it will return error message

Remove kernel module:

rmmod ip_tables

Remove module with wait util become unused:

rmmod -w ip_tables


Links


Modalias


Links
Configure hardware settings

 101.2 Boot the system


Boot sequence

MBR single OS

BIOS (Basic Input Output Service), stored on non-volatile ROM,EEPROM
BIOS after turn on, power-on self test (POST)
Loads first sequence from MBR (Master Boot Record)
MBR contains the partition table
The code loaded by BIOS from first sector is first stage boot loader
Check for primary partition marked as active, partition boot record
Load the OS, this is stage 2 boot loader
Notice: MBR system cannot boot from logical partition, only primary and active


Common linux bootloaders

LILO, the LInux LOader
GRUB, GRand Unified Boot loader (GRUB legacy)
GRUB2
Syslinux


Chain loading

boot manager could load another boot manager
e.g LILO in one partition chain load GRUB in another partition


EFI and UEFI

UEFI, Unified Extensible Firmware Interface
32bit, 64bit
Many, not all UEFI impl. allow booting in UEFI mode or Legacy


GRUB2 Menu

e - edit
c - grub command line


Kernel parameters

single user mode


The init process

when kernel finish loading, it usually starts /sbin/init
int always assigned to PID 1
System V / Upstart, /sbin/init
systemd, /sbin/init -> /lib/systemd/systemd


System V

runlevels

0     : poweroff (halt)
1     : rescue (single user mode), aliased as s or S
2     : multi-user mode without networking
3     : multi-user with networking
5     : multi-user with networking and the X Window System
6     : reboot


/etc/rc.d/init.d, /etc/init.d
after completion, init start getty in case of headless/console mode, or graphical login screen


Systemd targets (runlevels)
/usr/lib/systemd/system/runlevel0.target -> poweroff.target
/usr/lib/systemd/system/runlevel1.target -> rescue.target
/usr/lib/systemd/system/runlevel2.target -> multi-user.target
/usr/lib/systemd/system/runlevel3.target -> multi-user.target
/usr/lib/systemd/system/runlevel4.target -> multi-user.target
/usr/lib/systemd/system/runlevel5.target -> graphical.target
/usr/lib/systemd/system/runlevel6.target -> reboot.target


Init RAM disk

to list ram disk (CentOS 7), lsinitrd /boot/initramfs-3.10.0-693.11.6.el7.x86_64.img


Boot events

dmesg, kernel ring buffer

dmesg | head -n 30


kernel ring buffer is used also after system booted, e.g hot plug events, program failures


Logging systems messages

System V init, /var/log/messages
systemd, journalctl -xb (display entries from the current boot)


Links


how-to-use-journalctl-to-view-and-manipulate-systemd-logs


 101.3 Change runlevels / boot targets and shutdown or reboot system


System V

runlevels, look at 101.2 Boot the system
default runlevel is determined from id: , /etc/inittab

# Default runlevel. (Do not set to 0 or 6)
id:3:initdefault:


Changing runlevels

pernament, through /etc/inittab
boot, grub e.g pass 3 to kernel params
immediatelly telinit, /sbin/telinit -> init


Explore runlevel

runlevel
runlevel
N 3

N - previous runlevel, N means runlevel has not been changed since the system was booted
3 - current runlevel


Single-user mode

telinit 1, telinit s


Clear shutdown

shutdown [OPTIONS...] [TIME] [WALL...]
shutdown command, it shutdown or reboot system, the init process sends all running processes SIGTERM signal and
after 5 seconds delay init sends a SIGKILL signal
by default shutdown switches to runlevel 1, or to runlevel specified by -h halt or -r reboot
[TIME] e.g 23:59, relative 10m, immediate +0 alias for now
[WALL] message to be displayed 15 min before shutdown command is executed, if [TIME] is < 15 min then message is displayed immediately
-t60 delay SIGKILL signal after SIGTERM signal sent to processes, see above
shutdown -c cancel the shutdown command
if [TIME] is used, 5 minutes before the system goes down the /run/nologin file is created to ensure that further logins shall not be allowed.


Notify users with wall
$ wall Scheduled outage at 23:59 has been canceled

    Scheduled outage at 23:59 has been canceled
 
$ echo -e "We are experiencing system problemsOutage rescheduled to 02:30" | wall

     We are experiencing system problemsOutage rescheduled to 02:30


Halt, reboot, and poweroff

halt
poweroff, is symbolic link to halt
/sbin/poweroff -> halt


reboot, is symbolic link to halt
/sbin/reboot -> halt


System V /etc/inittab

/etc/inittab
id:runlevels:action:process
action:

respawn (getty)
wait
once
initdedault
ctrlaltdel


e.g
c1:12345:respawn:/sbin/agetty 38400 tty1 linux

ca::ctrlaltdel:/sbin/shutdown -t5 -r now


System V Initialization scripts

runlevels in /etc
rc -> rc.d/rc
rc0.d -> rc.d/rc0.d
rc1.d -> rc.d/rc1.d
rc2.d -> rc.d/rc2.d
rc3.d -> rc.d/rc3.d
rc4.d -> rc.d/rc4.d
rc5.d -> rc.d/rc5.d
rc6.d -> rc.d/rc6.d
rc.d/rc.local
rc.d/rc.sysinit


(centos 5) scripts are implemented in /etc/init.d/
script for runlevel is than a symbolic link to /etc/rc[0-6].d/ with pattern:

Snn (e.g. S27auditd) - for starting
Knn (e.g. K72auditd) - for switch off the service


cd /etc/rc5.d

S27auditd -> ../init.d/auditd

cd /etc/rc0.d

S72auditd -> ../init.d/auditd


Upstart

driven by events
Events are used to trigger tasks or services, collectively known as jobs


Systemd

units, which can be grouped into control groups or cgroups

service units
socket units
device units
mount units
automount units
target unit
snapshot units


Links


kernel-parameters.txt
Arch Wiki kernel_parameters
how-to-configure-a-linux-service-to-start-automatically-after-a-crash-or-reboot-part-1-practical-examples

	102.3 Manage shared libraries


Static and dynamic linking

Statically linked executables, /sbin/sln
Dynamically linked executables, /bin/ln


The ldd command

displays shared object dependencies of executable if any exists
$ ldd /sbin/sln
not a dynamic executable

$ ldd /bin/ln
linux-vdso.so.1 (0x00007fff9c8c6000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fc5c7ae0000)
  /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fc5c80a8000)


Dynamic library configuration

/etc/ld.so.conf, configuration
/etc/ld.so.conf.d, partial configuration read by ld.so.conf
/etc/ld.so.cache, used by dynamic loader
if /etc/ld.so.conf//etc/ld.so.conf.d were changed them ldconfig must run to rebuild /etc/ld.so.cache
to display ld.so.cache
$ /sbin/ldconfig -p


Loading specific libraries

e.g support old dynamic shared library, use
$ export LD_LIBRARY_PATH=/usr/lib/oldstuff:/opt/libs


Links:
-software-interrupts-vs-system-calls


PATA Disks

Master - /dev/hda
Slave - /dev/hdb
Partitions - /dev/hda1, /dev/hda2


SCSI Disks

Hard drives /dev/sda,/dev/sdb,..
CD/DVD ROM /dev/scdx,/dev/srx
Tape /dev/stx,/dev/nstx


	102.4 Use Debian package management


Installing Debian packages
sudo apt-get install openjdk-7-jre-headless


Simulated or dry-run install
sudo apt-get install -s openjdk-7-jre-headless


Package locations

Packages resources
cat /etc/apt/sources.list


Additional Packages resources configuration
/etc/apt/sources.list.d


To synchronize local database of packages, should run befor every package install or update
apt-get update


Removing Debian packages
remove - remove package, but configuration files persists in system

apt-get remove -s openjdk-7-jre-headless


autoremove, will removes also installed dependencies when the are no longer required
apt-get autoremove -s openjdk-7-jre-headless

apt-get remove auto-remove -s openjdk-7-jre-headless


autoremove whithout package name will remove all unused packages
apt-get autoremove


purge, remove package and configuration files
apt-get purge


Updating Debian packages
prerequisite packages will also be upgraded

sudo apt-get upgrade ghostscript


Updating all packages or upgrading to a new distribution
upgrade all packages

sudo apt-get upgrade

upgrade distribution

sudo apt-get dist-upgrade


APT configuration
/etc/apt/apt.conf


Reconfiguring Debian packages

debconf, re-configure packages after they are installed
dpkg-reconfigure tzdata


debconf-show, displays current configuration
debconf-show tzdata

debconf-show adduser


Package status with dpkg

configuration wide /etc/dpkg/dpkg.cfg
configuration user ~/.dpkg.cfg
dpkg status of packages
$ dpkg -s gfortran tzdata

Package `gfortran' is not installed and no info is available.

Package: tzdata
Status: install ok installed
Multi-Arch: foreign
...


Packages and the files in them

to list files(including directories) of package
dpkg -L gcc

/.
/usr
/usr/bin
/usr/bin/c89-gcc
/usr/bin/c99-gcc
...


Find Package which contains a file
dpkg -S /usr/lib/libmagic.so.1

libmagic1: /usr/lib/libmagic.so.1


Alternatives

for packages like java, openJDK etc.
update-alternatives

ls -la /etc/alternatives

e.g. to display actual command link

update-alternatives --display java


Aptitude

ncurses package manager
C-T access to menu
/ search for package


Finding Debian packages
apt-cache search "linux loader"


Download only mode
apt-get install -d gfortran

than list in

ls -la /var/cache/apt/archives/gfo*

/var/cache/apt/archives/gfortran_4%3a4.6.3-1ubuntu5_amd64.deb

than list contents with

dpkg -c /var/cache/apt/archives/gfortran_4%3a4.6.3-1ubuntu5_amd64.deb | grep -i --color 'bin'

./usr/bin/
./usr/bin/x86_64-linux-gnu-gfortran -> gfortran-4.6
./usr/bin/gfortran -> gfortran-4.6


Links
- what-is-the-difference-between-apt-get-and-dpkg

 102.5 Use RPM and YUM package management


Installing RPM packages

package management system used for packaging in the Linux Standard Base (LSB)
three subgroups:

Querying and verifying packages
Installing, upgrading, and removing packages
Performing miscellaneous functions


RPM install doesn't resolve dependencies, it warn but they must be downloaded and installed

rpm -i gcc-gfortran-4.8.5-16.el7.x86_64.rpm
warning: gcc-gfortran-4.8.5-16.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
error: Failed dependencies:
gcc = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libgfortran = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libgfortran.so.3()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libmpc.so.3()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libmpfr.so.4()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libquadmath = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libquadmath-devel = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64


YUM install of RPM packages

YUM resolve dependencies automatically

yum install gcc-gfortran


Package locations

repositories /etc/yum.repos.d/
YUM configuration file /etc/yum.conf
repo file is by default divided to 3 sections:

normal packages
debug packages
source packages


yum clean, clean locally stored packages database
yum makecache, create locally stored packages database


Removing RPM packages

RPM remove test run
RPM system does not maintain information on packages that were automatically added

sudo rpm -e --test vim-minimal
error: Failed dependencies:
    vim-minimal is needed by (installed) sudo-1.8.19p2-11.el7_4.x86_64


YUM will offer to remove also dependent packages
sudo yum remove vim-minimal

Removing:
vim-minimal
Removing for dependencies:
sudo

Remove  1 Package (+1 Dependent package)


Upgrading RPM packages

RPM upgrade
Download:

cairo-1.14.2-1.fc21.x86_64.rpm
cairo-gobject-1.14.2-1.fc21.rpm

cd to:

rpm -Uvh *.rpm

Preparing...                          ################################# [100%]
Updating / installing...
1:cairo-1.14.2-1.fc21              ################################# [ 25%]
2:cairo-gobject-1.14.2-1.fc21      ################################# [ 50%]
Cleaning up / removing...
3:cairo-gobject-1.13.1-0.4.git337ab################################# [ 75%]
4:cairo-1.13.1-0.4.git337ab1f.fc21 ################################# [100%]

Where:
- U upgrade/install packages
- v verbose
- h display hashmarks as progress bar


Yum upgrade
yum update 'pop*'
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package poppler.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler.x86_64 0:0.26.2-9.fc21 will be an update
---> Package poppler-data.noarch 0:0.4.7-1.fc21 will be updated
---> Package poppler-data.noarch 0:0.4.7-2.fc21 will be an update
---> Package poppler-glib.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler-glib.x86_64 0:0.26.2-9.fc21 will be an update
---> Package poppler-utils.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler-utils.x86_64 0:0.26.2-9.fc21 will be an update
...


Querying RPM packages

RPM basic query
rpm -q vim-minimal

vim-minimal-7.4.160-2.el7.x86_64


RPM detailed query
rpm -qi vim-minimal

Name        : vim-minimal
Epoch       : 2
Version     : 7.4.160
Release     : 2.el7


yum info
yum info vim-minimal

Installed Packages
Name        : vim-minimal
Arch        : x86_64
Epoch       : 2
...


rpm --queryformat, to build custom query output
rpm --querytags, to list all tags supported of rpm version


RPM packages and files in them

RPM query files in package (package must be installed)
rpm -ql vim-minimal

/etc/virc
/usr/bin/ex
/usr/bin/rvi
/usr/bin/rview
...


RPM query configuration files in package (package must be installed)
rpm -qc vim-minimal

/etc/virc    


RPM query documentation files in package (package must be installed)
rpm -qd vim-minimal

/usr/share/man/man1/ex.1.gz
/usr/share/man/man1/rvi.1.gz
/usr/share/man/man1/rview.1.gz
/usr/share/man/man1/vi.1.gz
/usr/share/man/man1/view.1.gz
/usr/share/man/man1/vim.1.gz
/usr/share/man/man5/virc.5.gz    


RPM query files in package (not installed only downloaded)
rpm -qpl gcc-gfortran-4.8.5-16.el7.x86_64.rpm    

/usr/bin/f95
/usr/lib/gcc
/usr/lib/gfortran
/usr/lib/gcc/x86_64-redhat-linux


RPM query documentation files in package (not installed only downloaded)
rpm -qpd gcc-gfortran-4.8.5-16.el7.x86_64.rpm

/usr/share/info/gfortran.info.gz
/usr/share/man/man1/gfortran.1.gz
...


Querying all installed packages
List all packages sorted

rpm -qa | sort | more

Count all packages in system

rpm -qa | wc -l

Count all files in installed system packages

rpm -qal | wc -l


Querying which package owns a file
$ which vi

/usr/bin/vi

$ rpm -qf $(which vi)

vim-minimal-7.4.160-2.el7.x86_64


RPM dependencies

RPM query package dependencies (not installed, download only)
$ rpm -qpR gcc-gfortran-4.8.5-16.el7.x86_64.rpm

/bin/sh
/bin/sh
/sbin/install-info
/sbin/install-info
gcc = 4.8.5-16.el7
ld-linux-x86-64.so.2()(64bit)
...


RPM query package dependencies (package is installed)
$ rpm -qR vim-minimal

config(vim-minimal) = 2:7.4.160-2.el7
libacl.so.1()(64bit)
libacl.so.1(ACL_1.0)(64bit)
libc.so.6()(64bit)
...


Yum list package dependencies

package dependencies with specified package architecture

yum deplist $(rpm -q vim-minimal)

package: vim-minimal.x86_64 2:7.4.160-2.el7
  dependency: libacl.so.1()(64bit)
    provider: libacl.x86_64 2.2.51-12.el7
  dependency: libacl.so.1(ACL_1.0)(64bit)
    provider: libacl.x86_64 2.2.51-12.el7
  dependency: libc.so.6(GLIBC_2.15)(64bit)
    provider: glibc.x86_64 2.17-196.el7_4.2
  ...


package dependencies without specified package architecture (display x86_64, i686)
$ yum deplist vim-minimal

...
  dependency: libtinfo.so.5()(64bit)
    provider: ncurses-libs.x86_64 5.9-14.20130511.el7_4
  dependency: rtld(GNU_HASH)
    provider: glibc.x86_64 2.17-196.el7_4.2
    provider: glibc.i686 2.17-196.el7_4.2
...


only package names
$ yum deplist $(rpm -q gcc-gfortran) | awk '/provider:/ { print $2 }'|sort|uniq

bash.x86_64
gcc.x86_64
glibc.i686
glibc.x86_64
...


RPM what package provide libtinfo.so.5()(64bit)
$ rpm -q --whatprovides 'libtinfo.so.5()(64bit)'

  ncurses-libs-5.9-14.20130511.el7_4.x86_64


YUM what package provide libtinfo.so.5()(64bit)
$ yum whatprovides 'libtinfo.so.5()(64bit)'

ncurses-libs-5.9-13.20130511.el7.x86_64 : Ncurses libraries
Repo        : base
Matched from:
Provides    : libtinfo.so.5()(64bit)

ncurses-libs-5.9-14.20130511.el7_4.x86_64 : Ncurses libraries
Repo        : updates
Matched from:
Provides    : libtinfo.so.5()(64bit)


RPM package file integrity
Package must be downloaded

$ rpm -vK gcc-gfortran-4.8.5-16.el7.x86_64.rpm

gcc-gfortran-4.8.5-16.el7.x86_64.rpm:
  Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
  Header SHA1 digest: OK (ae9044222237b6edb5c95e1dea593a7e99683596)
  V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
  MD5 digest: OK (15d89eebe4773f4c2e134636967901fb) 
  
NOKEY - means we do not have imported public key in RPM database


to verify package against signature, we need to:

download a package signature file
check the fingerprint
rpm --import the key


Links:


how-do-i-tell-which-gpg-key-an-rpm-package-was-signed-with


Verifying an installed package


to verify if installed package is not modified
$ rpm -V vim-minimal

empty if ok


if something missing e.g
$ sudo rm -v /usr/bin/vi
removed ‘/usr/bin/vi’

rpm -V vim-minimal
missing     /usr/bin/vi


to fix inconsistency
sudo yum reinstall vim-minimal


Force uninstall package with dependencies
rpm -e --nodeps vim-common

--nodeps bypass the dependency checking


Downloading RPMs from repositories
Will download only the requested package and its dependencies

$ yumdownloader --resolve gcc-gfortran.x86_64

after:

cpp-4.8.5-16.el7_4.2.x86_64.rpm
gcc-4.8.5-16.el7_4.2.x86_64.rpm
gcc-gfortran-4.8.5-16.el7_4.2.x86_64.rpm
...


Using rpm2cpio to unpack RPM package
rpm2cpio ../gcc-gfortran-4.9.2-6.fc21.x86_64.rpm | cpio -idv


Links


yum-download-mirror-switching-to-the-fastest-mirror-on-centosfedorarhel
how-to-make-yum-also-install-debug-symbols

 103.1 Work on the command line


Bash shell

buildin commands, cd,break,exec
external commands
use 3 standard I/O streams

stdin
stdout
stderr


prompt for root mostly has #, but depend on configuration
[root@f28 ~]#


Commands and sequences

Echo
$ echo Hello
Hello

$ echo A sentence
A sentence

$ echo A   sentence
A sentence

$ echo "A   sentence"
A sentence


Escapes and line continuation
$ echo -e "No new line\c"
No new line $
  
$ echo "A line with a typed
$ > return"
A line with a typed
> return

$ echo -e "A line with an escaped\nreturn"
A line with an escaped
return

$ echo "A line with an escaped\nreturn but no -e option"
A line with an escaped\nreturn but no -e option

$ echo -e Doubly escaped\\n\\tmetacharacters
Doubly escaped
      metacharacters


Bash shell metacharacters and control operators

metacharacters, | & ; ( ) < >
metacharacters must be either quoted or escaped using a backslash ()
control operators, || && & ; ;; | ( )
command sequences, non zero failure, zero success
$ echo line 1;echo line 2; echo line 3
line 1
line 2
line 3

$ echo line 1 && echo line 2 && echo line 3
line 1
line 2
line 3

$ echo line 1 || echo line 2; echo line 3
line 1
line 3


Exit
$ (echo In subshell; exit 0) && echo OK  || echo Bad exit
In subshell
OK

$ (echo In subshell; exit 4) && echo OK  || echo Bad exit
In subshell
Bad exit


Environment variables

to reference shell variables prefix with $name
common bash environment variables
$ echo $PPID - show parent process PID

$ echo $$ - show current process PID, in this case shell process

$ echo $? - exit code of the last command


other e.g
$ (exit 0);echo $?;(exit 4);echo $?
0
4


variables are case sensitive so, var1 != VAR1 different variables
technically $$ and $? aren't shell variables, rather parameters, you can't assing value to them
create shell variable, without visibillity to child shell e.g
$ VAR1=var1
$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1

- it's empty


create shell variable, with visibillity to child shell e.g
$ export VAR1=var1

$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1
var1


shell variable from child shell process is not visible to parent shell e.g
$ export VAR1=var1

$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1
var1

- export VAR2 in child shell
  $ export VAR2=var2

  $ echo $VAR2
  var2

- exit from child shell
  
  $exit
  
  $ ps -p $$ -o "pid ppid cmd"

  PID  PPID CMD
  2433  2406 -bash
  
  $ echo $VAR2
  - it's empty


expansion double quotes(")/(single quotes(') e.g
$ echo "$SHELL" '$SHELL' "$$" '$$'
/bin/bash $SHELL 2597 $$

double quotes("), expansion is done before string si passed
$ bash -c "echo Expand in parent $$ $PPID"
Expand in parent 2597 2571

single quotes('), expansion is done after string si passed, so in child shell
$ bash -c 'echo expand in parent $$ $PPID'
Expand in parent 12109 2597


using curly braces with variable names
$ echo "-$HOME/abc-"
-/home/bodo/abc-

$ echo "-$HOME_abc-"
--

$ echo "-${HOME}_abc-"
-/home/bodo_abc-


Env

-i, start with an empty environment

$ env
LANG=en_US.UTF-8
GDM_LANG=en_US.UTF-8
HISTCONTROL=ignoreboth
TODO=t
DISPLAY=:1
COLORTERM=truecolor
USERNAME=bodo
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins

$ env -i bash -c 'echo $SHELL; env'
/bin/bash
PWD=/home/bodo
SHLVL=1
_=/usr/bin/env


-i, start with an empty environment and -l login shell

$ env -i bash -lc 'echo $SHELL; env'
/bin/bash
LANG=en_US.UTF-8
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins
PWD=/home/bodo
XDG_DATA_DIRS=/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl


Unset and set

login shell
non-login shell
to display current shell options
$ echo $-
himBHs


set shell option
$ set -u
himuBHs


unset shell option
$ set +u
himBHs


e.g
$ echo $VAR1
var1

$unset VAR1;echo $VAR1
- empty

- causes bash to report an error with undefined variables rather than treat them as defined but empty
$ set -u;echo $VAR1
-bash: VAR1: unbound variable


Exec

replaces the current shell
$ echo $$
2597

$ bash
$ echo $$
8230

$ exec bash
$ echo $$
8230

$ exit

$ echo $$
2597


System information with uname

Links:


System information with uname


Command history

history for commands on/off
On - this is default
$ set -o

Off
$ set +o


history command
$ history
3639  bash
3640  echo $$
3641  uname
3642  uname -s
3643  uname -n

- display last 2 commands
$ history 2
3660  man 4 tty
3661  history 2

- to delete command
$ history -d 3661
 3660  man 4 tty
 3661  history -d 3661
 3662  history 3
 
- run most recent command (also !-1)
$ !!
history 3
  3660  man 4 tty
  3661  history -d 3661
  3662  history 3

- with replacing a command parameter
$ env -i bash -c 'echo $$'
26085
$ !ec:s/$$/$PPID/
echo $PPID
2571


logout, the last $HISTSIZE line are saved to ~/.bash_history, multiple sessions could rewrite the .bash_history


PATHS

determined with $PATH variable
e.g how to find a command executable with which or type

$ which bash env zip xclock echo set ls
/usr/bin/bash
/usr/bin/env
/usr/bin/zip
which: no xclock in (...
which: no set in (...
/usr/bin/ls

$ type bash env zip xclock echo set ls
bash is hashed (/usr/bin/bash)
env is hashed (/usr/bin/env)
zip is /usr/bin/zip
-bash: type: xclock: not found
echo is a shell builtin
set is a shell builtin
ls is aliased to `ls --color=auto'   


absolute path /bin/echo
relative path
$ cd /usr

$ ../bin/echo Relative path
Relative path


Changing your working directory
- display the path output when changed, in this example in /usr
export CDPATH=/usr:$CDPATH

$ cd /usr
$ cd local/
/usr/local/

- change to previous
$ cd -
/usr
$ cd -
/usr/local

- a previous path is stored in OLDPWD shell variable
$ echo $OLDPWD
/usr


Manual (man) pages

(1) User commands (ls,
(2) System calls or kernel function (link, mkdir)
(3) Library routines (btee, locale)
(4) Devices (tty)
(5) File format descriptions (keymaps, motd)
(6) Games
(7) Miscellaneous (arp,boot,regex)
(8) System administration (fsck, mount)

- device
$ man 4 tty

- user command to print device name to stdout
$ man 1 tty

- man pages with name
$ whatis tty
tty (1)              - print the file name of the terminal connected to standard input
tty (1p)             - return user's terminal name
tty (4)              - controlling terminal

- apropos, search for keyword in man pages
$ apropos tty
__gnu_parallel::__binder1st (3) - Similar to std::binder1st, but giving the argument types explicitly
agetty (8)           - alternative Linux getty
CMS_get0_eContentType (3ssl) - get and set CMS content types and content
CMS_set1_eContentType (3ssl) - get and set CMS content types and content


 103.3 Perform basic file and directory management


Listing directories

PWD shell variable


Listing details

On a storage device, a file or directory is contained in a collection of blocks
Information about a file is contained in an inode
inode number is also a file serial number it's unique whitina particular file system
ls doesn't list special files with (.)
-l --format=long

$ ls -la

-i list inodes numbers

$ ls -i

-d list directory itself not the contents

$ ls -ld

Sorting:
-t sorting by modification time
-S sorting by size

$ ls -lr
$ ls -lS

-r reverses

$ ls -ltr     


Links


why-size-reporting-for-directories-is-different-than-other-files


Copying, moving, and deleting files

cp, copy
cp -p preserve attributes (modifi time, own ...etc)
$ cp -p text2 text2.bak_3

-rwxrwxr-x  1 bodo users    6 Mar 18 21:12 text2
-rwxrwxr-x  1 bodo users    6 Mar 18 21:12 text2.bak_3


mv, move/rename
rm, remove


Rmdir

you can remove only empty directories
mkdir -p dir2/dir3

$ rmdir dir2
rmdir: failed to remove 'dir2': Directory not empty

$rmdir dir2/dir3
$rmdir dir2


Recursive manipulation

copy recursively e.g.
cp -pR . copy1


Wildcards and globbing

man 7 glob

? - matches any single character
* - matches any string, including an empty string
[0-9] - character class


Touching files

touch updates modification time
$ touch f1

$ stat f1
modification time should change


touch with -c (or --no-create)
$ touch -c f3

$ ls -l f3
- empty


touch with timestamp -t
$ touch -t 201408121510.59 f3
-rw-r--r-- 1 bodo users 0 Aug 12  2014 f3


touch with datestring -d
touch -d 11am f4
-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4


touch with reference, means it uses date a timestamp of ref. file
-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4

$ touch -r f4 f4a

$ ls -l f4*

-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4
-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4a


Finding files

find with name
$ find . -name "*[1k]*"
./f1
./dir_1
./dir_1/new_text1
./text1
./text[*?!1]
./backup
./backup/text1.~2~
./backup/text1
./backup/text1.~1~


find with wholename
$ find . -wholename "*ack*1"
./backup/text1


find with ignorecase
$ find . -iwholename "*ACK*1"
./backup/text1


find a directories
$ find . -type d
.
./dir_1
./backup


find by specifics size (units, default 'b' 512 bytes block, 'c' bytes, 'k' kilobytes, 'M' megabytes)
- empty size files

-rw-r--r-- 1 bodo users 0 Aug 12  2014 f3a

$ find . -size 0
f3a

- find size between, ignore *back* pattern

-rw-r--r-- 2 bodo users   19 Mar 18 20:51  text1
-rw-r--r-- 1 bodo users    5 Mar 18 22:05 'text[*?!1]'
-rw-r--r-- 1 bodo users    6 Mar 18 21:12  text2
-rw-r--r-- 1 bodo users    5 Mar 18 22:05 'text[2*?!]'

$ find . ! -iwholename '*back*' -size +4c -size -20c -print
./text2
./text1
./text[*?!1]
./text[2*?!]


find with action -ls, equivalent to ls -lids
$ find . ! -iwholename '*back*' -size +4c -size -20c -ls
 18613131      4 -rw-r--r--   1  bodo     users           6 Mar 18 21:12 ./text2
 18613130      4 -rw-r--r--   2  bodo     users          19 Mar 18 20:51 ./text1
 18613144      4 -rw-r--r--   1  bodo     users           5 Mar 18 22:05 ./text[*?!1]
 18613145      4 -rw-r--r--   1  bodo     users           5 Mar 18 22:05 ./text[2*?!]


find with -exec
$ find . ! -iwholename '*back*' -size +4c -size -20c -exec ls -l '{}' \;
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 ./text2
-rw-r--r-- 2 bodo users 19 Mar 18 20:51 ./text1
-rw-r--r-- 1 bodo users 5 Mar 18 22:05 './text[*?!1]'
-rw-r--r-- 1 bodo users 5 Mar 18 22:05 './text[2*?!]'

- remove empty files
$ find . -empty -exec rm '{}' \;

- rename all *.htm files to *.html
$ find . -name "*.htm" -exec mv '{}' '{}l' \; 


find within time range
-mtime -1 , less than 1*24
-mtime +1 , more than 1*24
- daystart, mean day start at 24:00, not relatively as mtime


Link:


find-difference-with-mtime-and


Identify files

/usr/bin/file
$ file /bin/echo
/bin/echo: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=f2aeec2985ecd663ef339313e6b0167722926e47, stripped


Compressing files

Using gzip and gunzip
Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ gzip text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.gz

Decompress:

$ gzip -d text2.gz

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2


Using xz and unxz
Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ xz text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.xz

Decompress:

$ unxz text2.xz

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2


Using bzip2 and bunzip2
Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ bzip2 text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.bz2

Decompress:

$ bunzip -d text2.bz2

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2


compress uning stdout
gzip:

$ cat text2 | gzip -c > text2.gz

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.gz

xz:

$ xz - < text2 > text2.xz

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.xz


Using tar

diff archive
$ cd ../lpi103-2

$ ls -l
-rw-r--r--  1 bodo users    4 Mar 19 21:17  f1

$ tar -cvzf lpitar1.tar.gz .

$ ls -l lpitar1.tar.gz
-rw-r--r-- 1 bodo users 795 Mar 21 21:08 lpitar1.tar.gz

$ date
Wed Mar 21 21:21:04 CET 2018

$touch f1
-rw-r--r-- 1 bodo users 4 Mar 21 21:08 f1

$ tar --diff --file lpitar1.tar.gz .
./f1: Mod time differs


list archive with filtering
$ tar -tvf lpitar1.tar.gz --wildcards '*f*  
-rw-r--r-- bodo/users        0 2014-08-12 15:10 ./f3a
-rw-r--r-- bodo/users        0 2018-03-19 11:00 ./f4
-rw-r--r-- bodo/users        4 2018-03-19 21:17 ./f1
-rw-r--r-- bodo/users        0 2014-08-12 15:10 ./f3
-rw-r--r-- bodo/users        0 2018-03-19 11:00 ./f4b


Using cpio


 103.4 Use streams, pipes, and redirects


Redirecting standard IO


n > - redirect to file descriptor, content lost without warnning


n >> - redirect to file descriptor, content append to end
$ ls
text1  text2  text3  xaa  xab  yaa  yab

$ ls x* z*
ls: cannot access 'z*': No such file or directory
xaa   xab

$ ls x* z* >stdout.txt 2>stderr.txt

$ ls w* y*
ls: cannot access 'w*': No such file or directory
yaa   yab

$ ls w* y* >> stdout.txt 2>>stderr.txt

$ cat stdout.txt
xaa
xab
yaa
yab

$ cat stderr.txt
ls: cannot access 'z*': No such file or directory
ls: cannot access 'w*': No such file or directory


set -o noclobber, prevent overwriting of file (set off set +o noclobber)
set -o noclobber

$ ls w* y* > stdout.txt 2>stderr.txt
-bash: stdout.txt: cannot overwrite existing file

Temporarily enable overwrite
$ ls w* y* >|stdout.txt 2>|stderr.txt

$ cat stdout.txt
yaa
yab

$ cat stderr.txt
ls: cannot access 'w*': No such file or directory


redirections, both 1 (stdout) and 2 (stderr)
$ ls x* z* &>output.txt


redirections, both 1 (stdout) and 2 (stderr) explicit
$ ls x* z* >output.txt 2>&1


Links:


bash-hackers redirection_tutorial


to /dev/null
stderr to /dev/null

$ ls x* z* 2>/dev/null


Redirecting input

input redirection
$ cat text1
1 apple
2 pear
3 banana

$ tr ' ' '\t' < text1
1    apple
2    pear
3    banana


input redirection with here-document
$ sort -k2 <<END
> 1 apple
> 2 pear
> 3 banana
> END
1 apple
3 banana
2 pear


here-document with tabs
$ cat here-document.sh

cat <<- EOF
apple
EOF
    cat <<- EOF
    pear
EOF

$ . here-document.sh
apple
pear


here-document without tabs
$ cat here-document.sh

cat << EOF
apple
EOF
    cat << EOF
    pear
EOF

$ . here-document.sh
apple
    pear


Creating pipelines

piping stdout to stdin, when using hyphen(-) in place of filename it's indicating input should come from stdin rather than file
$ tar -tvf lpi103_4.tar.gz .

$ mkdir new

$ gzip -c -d lpi103_4.tar.gz | tar -xvf - -C new

$ ls new
ex-here.sh  stderr.txt  stdout.txt  text1 ...


Using output as arguments

xargs command
$ cat text1
1 apple
2 pear
3 banana

$ xargs < text1
1 apple 2 pear 3 banana

$ xargs < text1 echo "arg list:"
arg list: 1 apple 2 pear 3 banana

$ xargs --max-args 3 <text1 echo "arg list:"
arg list: 1 apple 2
arg list: pear 3 banana

$ xargs -n 1 <text1 echo "arg list:"
arg list: 1
arg list: apple
arg list: 2
arg list: pear
arg list: 3
arg list: banana


using xargs with quoting
$ echo '"4 plums"' | cat text1 - | xargs -n 1
1
apple
2
pear
3
banana
4 plums


using xargs with lines of input
Replace:

$ xargs -I XYZ echo "START XYZ REPEAT XYZ END" <text1
START 1 apple REPEAT 1 apple END
START 2 pear REPEAT 2 pear END
START 3 banana REPEAT 3 banana END

$ xargs -IX echo "<X><X>" <text2
<9      plum><9 plum>
<3      banana><3       banana>
<10     apple><10       apple>

$ cat text1 text2 | xargs -L2
1 apple 2 pear
3 banana 9 plum
3 banana 10 apple


using xargs with lists of files
$ ls |xargs grep "1"
text1:1 apple
text2:10    apple
xaa:1 apple
yaa:1


show limits, in old kernels xargs max. number of argumets was limited
$ xargs --show-limits
Your environment variables take up 1691 bytes
POSIX upper limit on argument length (this system): 2093413


Using the find command with the -exec option or with xargs
With exec (calling for each found file, so so text3 -> text1 -> text3 -> text2):

$ find text[12] -exec cat text3 "{}" \;
This is a sentence.  This is a sentence.  This is a sentence. 
1 apple
2 pear
3 banana
This is a sentence.  This is a sentence.  This is a sentence. 
9       plum
3       banana
10      apple

With xargs (calling all together, so text3 -> text1 -> text2):

$ find text[12] | xargs -t cat text3
This is a sentence.  This is a sentence.  This is a sentence. 
1 apple
2 pear
3 banana
9       plum
3       banana
10      apple


using find and xargs with blanks in file names
There is "text 1" copy.  
-print0 option with find if you are piping the output to xargs, 
 and use the -0 option to tell xargs to expect null-delimited input.

$ find . -name "*1" -print0 |xargs -0 grep "1"
./text1:1 apple
./text 1:1 apple

same as,

$ grep "1" "text 1" text1
./text1:1 apple
./text 1:1 apple


Links


xargs-examples


Splitting output

tee command, duplicate the output to file

$ ls text[1-3] | tee f1 f2

$ cat f1
text1
text2
text3

$ cat f2
text1
text2
text3


 103.5 Create, monitor, and kill processes


Manage foreground and background jobs

process.sh script
process.sh

#!/bin/bash

echo "Begin of count process"

for i in $(seq 1 1000); do
     echo "i: $i"
     sleep 5
done

echo "End of count process"

exit 0


fg command
$ ./process.sh
i: 1
i: 2
...

To stop process

Ctrl+Z

[1]+  Stopped                 ./process.sh

$ jobs -l
[1]+  3005 Stopped                 ./process.sh

Bring process foreground again

$ fg %1
i: 3
...

Stop again

Ctrl+Z

[1]+  Stopped                 ./process.sh

$ fg %?pro
i: 4
...


put process to backgroud
$ ./process.sh
i: 1
i: 2
...

To stop process

Ctrl+Z

[1]+  Stopped                 ./process.sh

To background:

$ bg

To stop again

$ fg

Ctrl+Z

[1]+  Stopped                 ./process.sh


Standard IO and background processes

if stdin is requested and run in background, then bash will stop the process
$ (date; cat - > bginput.txt; date) &
[1] 19728
Sat Mar 24 20:19:28 CET 2018

[1]+  Stopped                 ( date; cat - > bginput.txt; date )

$ fg
( date; cat - > bginput.txt; date )
ok
and back

Ctrl + D

$ cat bginput.txt
ok
and back    


Run a process after log out

nohup
$ nohup ./process.sh 1 > job_1.txt 2>&1 &
[1] 12646


Monitor process

ps command
$ nohup ./process.sh 1 > job_1.txt 2>&1 &
$ nohup ./process.sh 1 > job_1.txt 2>&1 &
$ nohup ./process.sh 1 > job_1.txt 2>&1 &

[2]  16124 Running                 nohup ./process.sh 1 > job_1.txt 2>&1 &
[3]- 16220 Running                 nohup ./process.sh 2 > job_1.txt 2>&1 &
[4]+ 16316 Running                 nohup ./process.sh 3 > job_1.txt 2>&1 &

$ jobs -p
16124
16220
16316

$ jobs -p | xargs ps -F -p
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
bodo     17102  2074  0  3494  3388   2 21:14 pts/1    S      0:00 /bin/bash ./process.sh 1
bodo     17164  2074  0  3494  3376   0 21:14 pts/1    S      0:00 /bin/bash ./process.sh 2
bodo     17227  2074  0  3494  3336   6 21:14 pts/1    S      0:00 /bin/bash ./process.sh 3


to display like tree like structure ps -j --forest
$ ps -j --forest

PID  PGID   SID TTY          TIME CMD
2074  2074  2074 pts/1    00:00:00 bash
23013 23013  2074 pts/1    00:00:00  \_ ps


free displays the amount of free and used memory in your system
Displays -t total memory in -g gigabytes
$ free -tg

Displays -t total memory in -m gigabytes every 10 seconds
$ free -tms 10


Select and sort processes for display

ps to display all processes with controlling terminals (-a all, -f full)
$ ps -af


ps to display all processes without controlling terminals
$ ps -xf


ps display for particular command and custom output (-C process command, -o customized output)
$ ps -C bash -o user,pid,tty,time,comm


ps sort with order (+) ascending, (-) descending
sid - process group session id

$ ps -aj --sort +sid;-comm


pgrep display process defined as regex
$ pgrep -af system*


top - interactive ps


Links


inspect-manage-processes-ps


Send signals to processes

list of Signals
$ kill -l

man 7 signal


send stop signal to job (same as sent by Ctrl + Z)
$ nohup ./process.sh 1 > job.txt 2>&1 &
[1] 4976

$ kill -s SIGSTOP %1
[1]+  Stopped                 nohup ./process.sh > job_1.txt 2>&1  


send continue signal to job
$ kill -s SIGCONT %1


sent terminatig signal
SIGTERM - 15 

$ kill -s SIGTERM %1
[1]+  Terminated              nohup ./process.sh > job_1.txt 2>&1

Alternative:

$ kill -15 %1
[1]+  Terminated              nohup ./process.sh > job_1.txt 2>&1


terminating process unconditionally SIGKILL
$  nohup ./process.sh > job_1.txt 2>&1 &
[1] 20320

$ kill -9 %1
[1]+  Killed                  nohup ./process.sh > job_1.txt 2>&1


Links:


list-of-kill-signals


Logout and nohup

after logout nohup processes continued to work but they are detached from terminal
In pts1:

$  nohup ./process.sh > job_1.txt 2>&1 &
[1] 20320

$ exit

Open new terminal:

$ jobs -l
- empty

$ ps -C process.sh
PID TTY          TIME CMD
31715 ?        00:00:00 process.sh

? - not attached to any terminal


 103.6 Modify process execution priorities


Using ps to find niceness
$ ps -l


Link


LPI Notices - Prioritizing Processes

 104.1 Create partitions and filesystems


MBR Partitions

limited to 4 primary partitions (1-4)
logical partitions >= 5 (gaps are not allowed)
backup MBR e.g., sfdisk -d /dev/sda > sda-backup.txt
restore MBR e.g., sfdisk -d /dev/sda < sda-backup.txt
MBR partitions type code:

0x0c(FAT)
0x05(old type of extended partitions)
0x07(NTFS)
0x0f(a newer type of extended partitions)
0x82(Linux swap)
0x83(Linux filesystem)


GPT Partitions

can define up to 128
gaps can occure, e.g. three partitions numbered 3,9,55
GPT supports partition type codes, it's 16-byte GUID (128 bits) GUID


LVM - logical volume management

access the logical volumes /dev/mapper (e.g. /dev/mapper/myvol-home)
/boot cannot reside in LVM when using a boot loader which does not support LVM (GRUB 2 only support it)


Links


(ArchWiki - LVM)[https://wiki.archlinux.org/index.php/LVM]


 104.3 Control mounting and unmounting of filesystems


Mounting filesystems

mount the filesystem on the device
mounting a device requires root permissions or sudo
mount point must exist before you mount anything over it
if mount point contains subdirectories, after mount they are not visible
to list what is mounted use mount without an parameters, similar info /proc/mounts, /etc/mtab
mounting a specific filesystem type mount -v vfat /dev/sda3 /dos


Mount options

mount options override default behaviour
remount already mounted filesystem read-only
mount -o remount,ro /dos


remount already mounted filesystem read-write
mount -o remount,rw /dos


remount will not complete successfully if any process has open files or directories in the filesystem you are remounting


Labels, UUIDs, and links

to find UUID and Label(if present) of device
/dev/sdc - usb key

# blkid /dev/sdc1
/dev/sdc1: LABEL="ARCH_201508" UUID="FC75-EDAF" TYPE="vfat" PARTUUID="052cf3ed-01"

# mkdir -p /mnt/ARCH_201508

To mount with LABEL
# mount LABEL="ARCH_201508" /mnt/ARCH_201508
# ls -l /mnt/ARCH_201508
arch
...

# umount /mnt/ARCH_201508

To mount with UUID
# mount UUID="FC75-EDAF" /mnt/ARCH_201508
# ls -l /mnt/ARCH_201508
arch  


to find symbolic link to /dev/sdc1 in /dev tree
$ find /dev -lname "*sdc1"
/dev/disk/by-label/ARCH_201508
/dev/disk/by-uuid/FC75-EDAF
/dev/disk/by-partuuid/052cf3ed-01
/dev/disk/by-path/pci-0000:00:14.0-usb-0:4:1.0-scsi-0:0:0:0-part1
/dev/disk/by-id/usb-Kingston_DataTraveler_108_001A4D5F1A5CBB5130000199-0:0-part1
/dev/block/8:33


Boot time and fstab


root= parameter in GRUB and LILO to tell the boot loader what filesystem should be mounted as root


set root in GRUB2


after mounting, initialization run mount -a to automatically mount set of filesystems defined in /etc/fstab
$ cat /etc/fstab
# /dev/sda1
UUID=1638d092-0700-4688-b342-ea3051f9ffa2       /               ext4            defaults,noatime,discard        0 1


fstab format:

file sysytem, LABEL, UUID
mount point
type, should aldo by auto, ext4, btrfs .etc
option:

ro, read only
rw, reda-write
noauto, not automatically mounted at boot time
user, non-root user is permitted to mount/unmout media, now part of udev
exec, to allow execute files from mounted system
noexec, to not allow execute files from mounted system
noatime, do not reccord access times


dump, consider ext2 a ext3 for backups filesystem 0 - off, 1 - on
pass, pass the order to check filesystem at boot time 0 - off, 1 on for root partition, 2 on for non root partition


Links


Arch Wiki - fstab


Unmounting filesystems

it isn't possible to umount if there are processes with opend files from that filesystem
lsof
USB is mounted in /mnt/ARCH_201508/

# umount /mnt/ARCH_201508/
umount: /mnt/ARCH_201508/: target is busy

# lsof +f -- /mnt/ARCH_201508/ | grep -i --color DIR
bash      11913 root  cwd    DIR    8,33     4096     1 /mnt/ARCH_201508
vim       14361 root  cwd    DIR    8,33     4096     1 /mnt/ARCH_201508

# fuser -uvm /mnt/ARCH_201508/
                     USER     PID    ACCESS COMMAND
/mnt/ARCH_201508:    root     kernel mount  (root)/run/media/bodo/ARCH_201508
                     root      11913 ..c..  (root)bash
                     root      14361 ..c..  (root)vim

# umount /mnt/ARCH_201508/
- should successfully umounted


Removable filesystems

to enable u/mount under user specify 'uid=1000'


Swap space

to display swap device use swapon -s or cat /proc/swaps
to customize output
$ swapon --show=NAME,TYPE,SIZE
NAME      TYPE      SIZE
/dev/sda3 partition  12G


Links


lsof-survival-guide


 104.5 Manage file permissions and ownership


User and groups

identify user currently logged whoami
identify user groups
$ id
uid=1000(bodo) gid=100(users) groups=100(users)

$ groups
users


File ownership and permissions

(r) read, (w) write, (x) execute
directories:

(r) enable list content of directory
(w) create or delete files in the directory
(x) allow enter directory and access subdirectories


create files in subdirectories, "root" 0400 (dr--------), subdirectory (0700) (drwx------)
"root"       - dr-------- 3 bodo users 4096 Mar 25 17:02 dir1
                                                         |_  
subdirectory - drwx------ 2 bodo users 4096 Mar 25 18:53 subdir1

$ touch dir1/subdir1/subdir1_script.sh
touch: cannot touch 'dir1/subdir1/subdir1_script.sh': Permission denied


create files in subdirectories, "root" 0200 (d-w-------), subdirectory (0700) (drwx------)
$ touch dir1/subdir1/subdir1_script.sh
touch: cannot touch 'dir1/subdir1/subdir1_script.sh': Permission denied


create files in subdirectories, "root" 0100 (d--x------), subdirectory (0700) (drwx------)
$ touch dir1/subdir1/subdir1_script.sh

$ ls -l dir1/subdir1/subdir1_script.sh
-rw-r--r-- 1 bodo users 0 Mar 25 19:05 dir1/subdir1/subdir1_script.sh


Other filesystem objects

(-) regular file
d  directory
l  symbolic link
c  character special device
b  block special device
p  FIFO pipe
s  Socket


The eleventh character

recent enhacenment, not in all distributions


Changing permissions

adding permissions to all
$ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh

$ chmod -v +x script.sh
mode of 'script.sh' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x)    


adding premissions selective
$ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh

$ chmod -v o+x script.sh
mode of 'script.sh' changed from 0644 (rw-r--r--) to 0645 (rw-r--r-x)

$ chmod -v g+x script.sh
mode of 'script.sh' changed from 0645 (rw-r--r-x) to 0655 (rw-r-xr-x)

$ chmod -v u+x script.sh
mode of 'script.sh' changed from 0655 (rw-r-xr-x) to 0755 (rwxr-xr-x)

Same:

$ chmod -v ugo+x script.sh
mode of 'script.sh' changed from 0655 (rw-r-xr-x) to 0755 (rwxr-xr-x)    


remove premissions selective
$ chmod -v ugo-x script.sh

$ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh


Octal permissions

0 ---
1 --x
2 -w-
3 -wx
4 r--
5 r-x
6 rw-
7 rwx


Access modes

suid, When an executable program has the suid access modes set, it will run as if it had been started by the file's owner, rather than by the user who really started it.
It's (s)

-rwsr-xr-x 1 root root 55416 Dec 16 17:03 /usr/bin/passwd


set a remove suid
Set:

$ chmod u+s script.sh
-rwsr-xrwx 1 bodo users    0 Mar 25 19:18 script.sh

Remove:
chmod u-s script.sh


set a remove guid
Set:

chmod g+s script.sh
-rwxr-srwx 1 bodo users    0 Mar 25 19:18 script.sh

Remove:
chmod g-s script.sh


Directory with sgid  any files or directories created in it will inherit the group ID of the directory
Sticky bit, if set only user or root can delete or unlink the file
$ chmod -v +t dir1
mode of 'dir1' changed from 0755 (rwxr-xr-x) to 1755 (rwxr-xr-t)


Immutable files
As root:

$ chattr +i /share/temp/lpi/lpi104-5/script.sh

$lsattr /share/temp/lpi/lpi104-5/script.sh
----i---------e--- /share/temp/lpi/lpi104-5/script.sh


The file creation mask

represents a numeric permission value to be removed

$ umask
022 (that is u=rwx,g=rx,o=rx)

In this case group, owner write permission


Links


Umask - description


 104.6 Create and change hard and symbolic links


Hardlinks
Softlinks
Identifying links
Who links to me?

finding hardlinks
$ find lpi104-6 -samefile lpi104-6/file1
lpi104-6/file3
lpi104-6/file1
lpi104-6/subdir/file3sub

$ ls -i lpi104-6/
18744944 file1

$ find lpi104-6 -inum 18744944
lpi104-6/file3
lpi104-6/file1
lpi104-6/subdir/file3sub  


finding softlinks
$ find lpi104-6 -lname "*file1"
lpi104-6/file4
lpi104-6/file6


Resources:


(IBM - A roadmap for LPIC-1)[https://www.ibm.com/developerworks/library/l-lpic1-map/]