Linux System Mining with Python

pylinux.rst

Update

The talk never happened, but I wrote an article on this topic: http://amitsaha.github.io/site/notes/articles/python_linux/article.html

Introduction

Most of the information that a power user or a sysadmin may need to know about a running Linux system is available in the form of plain text files. Utilities such as 'cat',' head', 'less' and 'grep' are sufficient to mine information that one may need from these files. Things however can get unwieldy when the task at hand demands writing shell scripts to couple these utilities. As much as power Linux users (including yours truly) would absolutely love to master shell scripting, the learning curve is quite steep and can be cryptic. This is certainly an opportunity to explore other ubiquitous alternatives for stitching together utility scripts. Bring in Python.

Overview

• Python is available on (almost) all Linux distributions

• Powerful text processing possible without even using any standard library modules (For eg. split() and list processing)

• File contents have a pattern (See 'cat /proc/cpuinfo' or 'cat /proc/net/dev)

• Identify patterns and harness them

• Example:

  >>> with open('/proc/version') as f:
  ...     data=f.readlines()
  ...
  >>> data
  ['Linux version 3.6.6-1.fc17.x86_64 (mockbuild@) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2)   (GCC) ) #1 SMP Mon Nov 5 21:59:35 UTC 2012\n']
  >>> data[0].split()[2]
  '3.6.6-1.fc17.x86_64'
  

• Complicated example (Reading Network device information)

  >>> with open('/proc/net/dev') as f:
  ...     data=f.readlines()
  
  >>> for line in data[2:]:
  ...     print line.strip().split(':')[0],line.strip().split(':')[1].split()[0],line.strip().split(':')[1].split()[8]
  ...
  lo 177921614534 177921614534
  em1 2199710944 155282249
  

• Standard modules: os, subprocess, sys, platform.

• When the information is not available easily via any of the files, use the subprocess module to execute native commands.

• Example:

  >>> import subprocess
  >>> for disk in subprocess.check_output(['df','-h']).split('\n'):
  ...     print disk
  ...
  Filesystem                   Size  Used Avail Use% Mounted on
  rootfs                        50G   30G   19G  62% /
  devtmpfs                     1.4G     0  1.4G   0% /dev
  tmpfs                        1.4G   96K  1.4G   1% /dev/shm
  tmpfs                        1.4G  5.6M  1.4G   1% /run
  /dev/mapper/vg_zion-lv_root   50G   30G   19G  62% /
  tmpfs                        1.4G     0  1.4G   0% /sys/fs/cgroup
  tmpfs                        1.4G     0  1.4G   0% /media
  /dev/mapper/vg_zion-lv_home   67G   32G   32G  50% /home
  /dev/sda5                    485M   99M  361M  22% /boot
  

• pylinux: Python interface to Linux System Information attempts to make system information available via a Python package.

• Giving an interface to your programs using optparse
• Making your scripts installable system wide
• Miscellaneous

Miscellaneous

• Takeaway: Fun with some text processing and in the process knowing more about your running Linux system. You don't need to be a shell scripting Ninja.
• Audience: Some Python programming and knowledge of Linux system internals required. You should know what you want to know.