Demystifying Jupyter Shell Variable Substitution SEO: magic %%bash %%sh bang ! OS

JupyterShellExperiments.txt

# TL;DR
# There is some tricky behavior lurking underneath Jupyter's shell variable substition syntax.
# For the best experience, stick with single quotes with {} variable placeholders, e.g:
python_variable='blah'
!echo somebashcommand '{python_variable} some extra gobblygook'

### Lessons Learned
# 1) Failed substitions (including unintentional variables) cause silent failures preventing any substitution.
# 1.1) If your substitution isn't working:
# 1.1.1) Prefix with echo, and strip your statement down until you find the part that isn't working
# 1.1.2) Investigate any intended literal $ or {} for escaping issues
# 2) Don't use single quotes with $, it doesn't substitute.
# 3) Jupyter substition operates the same inside single and double quotes, but the bash substition still treats them differently (double quotes may require bash escaping).
# 4) Stick with single quotes and {var} format for the least suprises, when possible

### Experiments
# define our python variable
hello = 'world'

### Observe the many ways to do the substitution into bash command
# Observe that substition of python variables works inside all manner of quotes, EXCEPT curiously '$var'
!echo 1 OK {hello}
!echo 2 OK $hello
!echo 3 OK "{hello}"
!echo 4 OK "$hello"
!echo 5 OK '{hello}'
!echo 6 BAD '$hello' # Surprise!
!echo 7 OK '''{hello}''' """{hello}"""

# 1 OK world
# 2 OK world
# 3 OK world
# 4 OK world
# 5 OK world
# 6 BAD $hello
# 7 OK world world

# And can even do evaluation
!echo 8 OK "{hello + ' record'}"

# 8 OK world record

### Gotcha: 
# Why does this not do our substitutions?
var1="value1"
var2="value2"
!echo 9 BAD mycommand -param1 {var1} -param2 {var2}  -o 'jsonpath={.data.\.dockerconfigjson}'

# 9 BAD mycommand -param1 {var1} -param2 {var2} -o jsonpath={.data.\.dockerconfigjson}

# Turns out that if you don't properly escape things, substition mysteriously breaks with no indications.
# Debugging technique => prefix with echo, start deleting stuff until it starts working, then figure out what's going on.
!echo 10 OK mycommand -param1 {var1}
!echo 11 OK mycommand -param1 {var1} -param2 {var2}
!echo 12 BAD mycommand -param1 {var1} -param2 {var2}  -o 'jsonpath={.data.\.dockerconfigjson}'

# 10 OK mycommand -param1 value1
# 11 OK mycommand -param1 value1 -param2 value2
# 12 BAD mycommand -param1 {var1} -param2 {var2} -o jsonpath={.data.\.dockerconfigjson}

# Maybe not so hard to figure out with this example, it's the '{}'
# Escape them by doubling them up
!echo 13 OK mycommand -param1 {var1} -param2 {var2}  -o 'jsonpath={{.data.\.dockerconfigjson}}'

# 13 OK mycommand -param1 value1 -param2 value2 -o jsonpath={.data.\.dockerconfigjson}

# What really appears to be happening is that `{.data.\.dockerconfigjson}` attempts an unintended 
# (and undefined) variable substitution which causes a silent failure (limitation of this technique)

# Dollar sign literal behavior
debtor = "fred"
!echo 14 BAD "$debtor owes me $10" # $10 causes failed substitution if not $$ escaped
!echo 15 BAD "$debtor owes me \$10"
!echo 16 BAD "{debtor} owes me $$10"
!echo 17 BAD '{debtor} owes me $$10'
!echo 18 OK "{debtor} owes me \$$10"
!echo 19 OK '{debtor} owes me $10'

# 14 BAD fred owes me 10
# 15 BAD fred owes me
# 16 BAD fred owes me 
# 17 BAD fred owes me $$10
# 18 OK fred owes me $10
# 19 OK fred owes me $10