Skip to content

Instantly share code, notes, and snippets.

@parente
Last active Jul 30, 2022
Embed
What would you like to do?
Jupyter Tidbit: IPython's ! returns an SList

Summary

IPython shell assignment (the ! operator) evaluates a command using the local shell (e.g., bash) and returns a string list (IPython.utils.text.SList). An SList is a list-like object containing "chunks" of stdout and stderr, properties for accessing those chunks in different forms, and convenience methods for operating on them.

Example

Binder

The SList.ipynb notebook below uses SList properties to access the output of a shell command as a list-like of strings, a newline-separated string, a space-separated string, and a list of pathlib.Path objects. The notebook then uses the SList.fields() and SList.grep() methods to extract columns from and search command output.

Why is this useful?

You can take advantage of the properties and methods of an SList to transform shell output into forms more amenable to further operations in Python.

For more information

See the IPython documentation about Shell Assignment for examples of executing shell commands with optional Python values as inputs. See the IPython documentation about String Lists for additional demontrations of the utility of SLists.

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Start with a simple `ls` command."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"ls = !ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the return type is not a simple `list`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"IPython.utils.text.SList"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(ls)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is an `SList`, a list-like object that contains \"chunks\" of `stdout` and `stderr`, properties for accessing those chunks in different forms, and convenience methods for operating on them."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['conda-bld', 'README.md', 'requirements.txt', 'SList.ipynb']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are many ways to refer to the output from an `SList` as a list-like of strings."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls == ls.get_list() == ls.list == ls.l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some properties also return the output as a newline delimited string."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"conda-bld\n",
"README.md\n",
"requirements.txt\n",
"SList.ipynb\n"
]
}
],
"source": [
"print(ls.nlstr)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls.get_nlstr() == ls.nlstr == ls.n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other properties return the output as a space separated string."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"conda-bld README.md requirements.txt SList.ipynb\n"
]
}
],
"source": [
"print(ls.spstr)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls.get_spstr() == ls.spstr == ls.s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Still other properties return the output as a list of `pathlib.Path` instances."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[PosixPath('conda-bld'),\n",
" PosixPath('README.md'),\n",
" PosixPath('requirements.txt'),\n",
" PosixPath('SList.ipynb')]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls.paths"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ls.get_paths() == ls.paths == ls.p"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pathlib\n",
"isinstance(ls.paths[0], pathlib.Path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are convenient for performing further path operations in Python."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[True, False, False, False]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[p.is_dir() for p in ls.paths]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`SList` objects expose a `fields()` method."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"df = !df -h"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['Filesystem Size Used Avail Use% Mounted on',\n",
" 'overlay 981G 577G 404G 59% /',\n",
" 'tmpfs 26G 0 26G 0% /dev',\n",
" 'tmpfs 26G 0 26G 0% /sys/fs/cgroup',\n",
" '/dev/sda1 981G 577G 404G 59% /etc/hosts',\n",
" 'shm 64M 0 64M 0% /dev/shm',\n",
" 'tmpfs 26G 0 26G 0% /sys/firmware']"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`fields` splits the output into whitespace delimited columns and returns the values of columns, specified by their indices, as space-separated strings."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['Filesystem Use%',\n",
" 'overlay 59%',\n",
" 'tmpfs 0%',\n",
" 'tmpfs 0%',\n",
" '/dev/sda1 59%',\n",
" 'shm 0%',\n",
" 'tmpfs 0%']"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.fields(0,4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`SList` objects also expose a `grep()` method. `grep` evaluates a regular expression or callable against all elements of the `SList` or a whitespace delimited column in each element."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['tmpfs 26G 0 26G 0% /dev',\n",
" 'shm 64M 0 64M 0% /dev/shm']"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.grep('dev', field=5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`grep(prune=True)` turns the grep into a filtering operation instead of a matching operation."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"hosts = !cat /etc/hosts"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Kubernetes-managed hosts file.\n",
"127.0.0.1\tlocalhost\n",
"::1\tlocalhost ip6-localhost ip6-loopback\n",
"fe00::0\tip6-localnet\n",
"fe00::0\tip6-mcastprefix\n",
"fe00::1\tip6-allnodes\n",
"fe00::2\tip6-allrouters\n",
"10.12.9.251\tjupyter-3675d82eae802db2c011037033d614a5-2dlwedcif3\n"
]
}
],
"source": [
"print(hosts.n)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The return value of `grep` (and `fields`) is another `SList` supporting all of the features noted above."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Kubernetes-managed hosts file.\n",
"127.0.0.1\tlocalhost\n",
"10.12.9.251\tjupyter-3675d82eae802db2c011037033d614a5-2dlwedcif3\n"
]
}
],
"source": [
"print(hosts.grep('ip6', prune=True).n)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
@cupdike
Copy link

cupdike commented Dec 3, 2021

Thanks, this was useful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment