Jupyter Tidbit: IPython's ! returns an SList

README.md

Summary

IPython shell assignment (the ! operator) evaluates a command using the local shell (e.g., bash) and returns a string list (IPython.utils.text.SList). An SList is a list-like object containing "chunks" of stdout and stderr, properties for accessing those chunks in different forms, and convenience methods for operating on them.

Example

The SList.ipynb notebook below uses SList properties to access the output of a shell command as a list-like of strings, a newline-separated string, a space-separated string, and a list of pathlib.Path objects. The notebook then uses the SList.fields() and SList.grep() methods to extract columns from and search command output.

Why is this useful?

You can take advantage of the properties and methods of an SList to transform shell output into forms more amenable to further operations in Python.

For more information

See the IPython documentation about Shell Assignment for examples of executing shell commands with optional Python values as inputs. See the IPython documentation about String Lists for additional demontrations of the utility of SLists.

SList.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start with a simple `ls` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls = !ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the return type is not a simple `list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IPython.utils.text.SList"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(ls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is an `SList`, a list-like object that contains \"chunks\" of `stdout` and `stderr`, properties for accessing those chunks in different forms, and convenience methods for operating on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['conda-bld', 'README.md', 'requirements.txt', 'SList.ipynb']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways to refer to the output from an `SList` as a list-like of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls == ls.get_list() == ls.list == ls.l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some properties also return the output as a newline delimited string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conda-bld\n",
      "README.md\n",
      "requirements.txt\n",
      "SList.ipynb\n"
     ]
    }
   ],
   "source": [
    "print(ls.nlstr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls.get_nlstr() == ls.nlstr == ls.n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other properties return the output as a space separated string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conda-bld README.md requirements.txt SList.ipynb\n"
     ]
    }
   ],
   "source": [
    "print(ls.spstr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls.get_spstr() == ls.spstr == ls.s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still other properties return the output as a list of `pathlib.Path` instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('conda-bld'),\n",
       " PosixPath('README.md'),\n",
       " PosixPath('requirements.txt'),\n",
       " PosixPath('SList.ipynb')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls.paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls.get_paths() == ls.paths == ls.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pathlib\n",
    "isinstance(ls.paths[0], pathlib.Path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are convenient for performing further path operations in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[True, False, False, False]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[p.is_dir() for p in ls.paths]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SList` objects expose a `fields()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = !df -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Filesystem      Size  Used Avail Use% Mounted on',\n",
       " 'overlay         981G  577G  404G  59% /',\n",
       " 'tmpfs            26G     0   26G   0% /dev',\n",
       " 'tmpfs            26G     0   26G   0% /sys/fs/cgroup',\n",
       " '/dev/sda1       981G  577G  404G  59% /etc/hosts',\n",
       " 'shm              64M     0   64M   0% /dev/shm',\n",
       " 'tmpfs            26G     0   26G   0% /sys/firmware']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fields` splits the output into whitespace delimited columns and returns the values of columns, specified by their indices, as space-separated strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Filesystem Use%',\n",
       " 'overlay 59%',\n",
       " 'tmpfs 0%',\n",
       " 'tmpfs 0%',\n",
       " '/dev/sda1 59%',\n",
       " 'shm 0%',\n",
       " 'tmpfs 0%']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.fields(0,4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SList` objects also expose a `grep()` method. `grep` evaluates a regular expression or callable against all elements of the `SList` or a whitespace delimited column in each element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tmpfs            26G     0   26G   0% /dev',\n",
       " 'shm              64M     0   64M   0% /dev/shm']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.grep('dev', field=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`grep(prune=True)` turns the grep into a filtering operation instead of a matching operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "hosts = !cat /etc/hosts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Kubernetes-managed hosts file.\n",
      "127.0.0.1\tlocalhost\n",
      "::1\tlocalhost ip6-localhost ip6-loopback\n",
      "fe00::0\tip6-localnet\n",
      "fe00::0\tip6-mcastprefix\n",
      "fe00::1\tip6-allnodes\n",
      "fe00::2\tip6-allrouters\n",
      "10.12.9.251\tjupyter-3675d82eae802db2c011037033d614a5-2dlwedcif3\n"
     ]
    }
   ],
   "source": [
    "print(hosts.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The return value of `grep` (and `fields`) is another `SList` supporting all of the features noted above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Kubernetes-managed hosts file.\n",
      "127.0.0.1\tlocalhost\n",
      "10.12.9.251\tjupyter-3675d82eae802db2c011037033d614a5-2dlwedcif3\n"
     ]
    }
   ],
   "source": [
    "print(hosts.grep('ip6', prune=True).n)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Thanks, this was useful!