Jenkinsfile idiosynchrasies with escaping and quotes

Jenkinsfile.groovy

node {
    echo 'Results included as an inline comment exactly how they are returned as of Jenkins 2.121, with $BUILD_NUMBER = 1'
    echo 'No quotes, pipeline command in single quotes'
    sh 'echo $BUILD_NUMBER' // 1
    echo 'Double quotes are silently dropped'
    sh 'echo "$BUILD_NUMBER"' // 1
    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"' // 1
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"' // "1"
    echo 'Using three backslashes still results in only preserving the quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"' // "1"
    echo 'To end up with \" use \\\\\\" (yes, six backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
    echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
    def foo = 'bar'
    sh 'echo "${foo}"' // (returns nothing)
    echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
    sh "echo \"${foo}\"" // bar
    echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
    sh 'echo \\\\\\"' + foo + '\\\\\\"' // \"bar\"
    echo 'We still needed all of that escaping and mixing concatenation is hideous!'
    echo 'There must be a better way, enter dollar slashy strings (actual term)'
    def command = $/echo \\\"${foo}\\\"/$
    sh command // \"bar\"
    echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
    def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
    sh vash // \"3\" bar
    echo 'It still requires escaping the escape but that is just bash being bash at that point'
    echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

I think "idiosynchrasies" is being overly polite!

thanks, you saved my day

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

Thank you for creating this, the \\\\\\\ saved me here!

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

Even with this guide I just spent 3+ hours with this :@

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

@laapsaap Lol!, Indeed i had one..

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }

[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }

For anyone that ends up here, the Snippet Generator is your best friend.

@Faheetah Nice gist ! Could you add %, that you have to double ? Like sh "echo '1 % 2'" doesn't display the %, you have to double it : "1 %% 2"

seeing this gist made me realized I am not alone on this pain "yay"

I am stuck with this for two friggin days :-/ .

sh 'docker exec tnd-chrome bash -c sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\'--tags @Regression\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'

this causes sudo to throw usage usage: sudo -h | -K | -k | -V etc...

		sh "docker exec tnd-chrome bash -c \\'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\\\'--tags @Regression\\\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify\\'"

this causes mvn: -c: line 0: unexpected EOF while looking for matching `''
mvn: -c: line 1: syntax error: unexpected end of file

:/ Any advice greatly appreciated

		sh "docker exec tnd-chrome bash -c 'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\\\'--tags @Regression\\\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'"

unexpected EOF while looking for matching `''

Tears of joy :') :')

This worked!!!

sh "docker exec tnd-chrome bash -c 'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\"--tags @Regression\" -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'"

The situation is much worse than this. Any space-separated arguments inside of quotes sent to Jenkins' awful sh task are broken up, single-quoted, and recombined against the mangled outer quoting. Christ, just pass the data through as it was input!

All of this works as expected, people.

The quotes are not dropped by anything in Jenkins. They are interpreted by your shell. Try running some of those commands in bash manually, and you'll see the same behaviour. That is,
echo a
and
echo "a"
will both output the single character a. When you see the shell command being run in the log (the lines that begin with a +), it is not Jenkins showing it to you before sending it to the shell; rather, it is the shell itself echoing it back, as started with the -x (aka -o xtrace) flag.

The backslashes on the other hand are being processed by both Jenkins and the shell. That is, when you write \\\\\\\" as you show above, Jenkins eats one set of escapes, and sends \\\" to the shell; then the shell does one more round, sending a literal \" to echo.

Note, however, that you almost never want this, most executables do not expect literal quotes in their parameters – those are for the shell.

You can easily test that things are working as intended with a shell script that lists its parameters line-by-line:

#!/bin/bash
echo "Got $# args:"
while [ $# -gt 0 ]; do
    echo "$1"
    shift
done
echo

Save this as something like printargs.sh and then replace every echo in your example with it.

Oh, and replace $BUILD_NUMBER with some variable that has spaces in its value, eg:

node {
    withEnv(['FOO=a b c']) {
        sh 'printargs.sh $FOO'
        sh 'printargs.sh "$FOO"'
    }
}

Quoth the @rudolfwalter:

All of this works as expected, people.

The quotes are not dropped by anything in Jenkins. They are interpreted by your shell.

Exactly right. Taking it line-by-line, starting at https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L4-L5 :

   echo 'Double quotes are silently dropped'
   sh 'echo "$BUILD_NUMBER"'

I'll use $USER as my example variable, since it's (pretty much) always present in Bash, whereas $BUILD_NUMBER only exists inside Jenkins runs. So, in Bash, this is equivalent to typing:

echo "$USER"

Which, as expected, generates just bmdan.

Next up, https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L6-L9 :

    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"'
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"'

What we're seeing here is Groovy's escaping. The rules are as follows:

1. After sh, look for the next string literal opener.
2. Load the string (honoring escapes), until you hit the corresponding (unescaped) closer for your opener.
3. Send the string to the system(3) or execve(2) call.

In the first case, Groovy helpfully honors a meaningless escape (\" inside of single quotes is just a literal ", same as it would have been without the \). Thus, after step 2, we're sending echo "$BUILD_NUMBER" to the shell, same as the previous two lines. In the second case, Groovy's escaping drops the \\ down to just a \, and sends that through, so the final command sent to the shell is echo \"$BUILD_NUMBER\". Try that as echo \"$USER\" in your shell and you'll see the same output: quotes around the var.

Moving on to https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L10-L11 :

    echo 'Using three backslashes still results in preserving the single quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"'

Three backslashes followed by a " now makes perfect sense, because Groovy sees this as two escapes: the first, an escaped backslash. The second, a (needlessly-) escaped ". So when this goes into the shell, it's echo \"$BUILD_NUMBER\", indistinguishable from the two slashes example above it.

Finally, we have https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L12-L13 :

    echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'

First off, no, you have six backslashes here. Which is exactly how many it should take. The final string sent to the shell is echo \\\"$BUILD_NUMBER\\\": an escaped \ and an escaped ", concatenated, on each side of the variable. It takes six backslashes to make sure that three backslashes end up at the shell once Groovy's done with it. Mind you, doing this is almost certainly not what you wanted to do; this prints out literal quotes and backslashes. If you're doing that in preparation for loading this into something else, you want to escape the contents of the variable, too, or something like echo \\\\\\"$PATH\\\\\\" is going to go straight to hell if there's an embedded " or \ inside $PATH.

Bonus: if you do happen to use seven backslashes, though, you get the exact same output. Think through why: any odd number of backslashes is simply going to hand some number of backslashes to the shell, and then escape the character following the last backslash in Groovy. Thus, final backslash will never show up by the time the shell gets it, the same as the 1- and 3-backslash formulations, above.

@michaelPf85 added, thanks!

I'm surprised this gist is still so active, good info everyone. I don't always see replies so best bet for more prompt feedback would probably be to post on SO or something, but I'll try to keep this updated with suggestions as I see them.

Edit: um, this doesn't look to be the case. When I use sh 'echo something with %%' it comes out with both of them.

@CH-DanReif I'm going to test run this and include the results as of Jenkins 2.121 since it should help a bit, and let me error check this (might have been updates since I wrote it). Looks like it is indeed six backslashes and I've updated the echo to note that, although seven technically works because of the aforementioned triple backslash. And yes there are some better ways of handling that but there are use cases for passing a literal "$var" to the shell. This gist doesn't reflect best bash practices but rather how to get around how clunky backslashes are going through several layers of escaping. It may not be a var but a string with a dollar sign in it too.

So how would I escape this dos batch command for a Jenkinsfile bat command?

%OC_EXE% -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test DNC.XTest\DNC.XTest.csproj --test-adapter-path:. --logger:xunit;LogFilePath=..\xunit-test-results.xml" -output:"%RAWOUTPUT%" -filter:+[*]* -oldStyle

slashy string don't seem to work, or I haven't nutted it yet.

In my Jenkinsfile I have this:
bat "${env.OC_EXE} -register:user -target:${env.DOTNET_EXE} -targetargs:'test --test-adapter-path:. --logger:xunit;LogFilePath=../${env.UT_RESULTS}' -output:../${env.RAWOUTPUT} -filter:+[*]* -oldStyle"

Which I can see in the console produces this:
G:\Jenkins\workspace\POC\Pipeline\DNCTest\DNC.XTest>C:\Users\xxx/.nuget/packages/opencover/4.6.519/tools/OpenCover.Console.exe -register:user -target:C:/PROGRA~1/dotnet/dotnet.exe -targetargs:'test --test-adapter-path:. --logger:xunit;LogFilePath=../xunit-test-results.xml' -output:../raw_coverage.xml -filter:+[*]* -oldStyle

But is throwing this error:
Incorrect Arguments: The argument '--test-adapter-path' is not recognised

How the heck do I correctly wrap -targetargs correctly so groovy will accept it and OpenCover knows to just pass --test-adapter-path onto dotnet.exe??

@agray this comes more down to how does Jenkins handle and escape spaces. This is a blight on any code trying to wrap shell commands, you come across it in any language and framework. Usually it's done with lists separating args so you get something like:

doShell(['ls', '-ahl', 'some directory'])

Which gets interpreted the same as running:

$ ls -ahl some\ directory

Jenkins, from all I've seen, has decided to use only strings as shell input. If you are dealing with too many types of things to escape, it might be easier to use an explicit script then call that. I haven't used Jenkins with Windows scripting but with Linux I believe you have to escape each quote or it disappears, so you would have to do something akin to:

sh 'foo --bar=\\\"something with spaces\\\"'

In my case the default shell happened to be 'dash', so a saner solution was to just switch to bash:

sh "#!/bin/bash\n" + """echo "with 'quotes'!" """

dollar-slashy should work too, and no excaping needed

Hello! Excuse me, when I use Blue Ocean, I want to extract the built-in variable $JOB_NAME of jenkins. For example, when extracting a-app job, I want job_name to be a-app, but the result of Blue Ocean is a-app/master, or a-app/dev, with an additional branch name. How should Jenkins file be written?

First time I've ever written a Jenkins pipeline and it may be the last ;)

This is driving me crazy. You can see below that I've created the variable projectName and used it successfully in many places. However when I use it in the 'git push' command it resolves to empty string. I've used ' quotes in this line. If I change it to " quotes, then the username & password variables don't seem to resolve properly.

Any suggestions?

def builder(projectName) {
    build job: "ESB-Relationship/${projectName}/OSDEBuild", parameters: [string(name: 'IMAGE_STREAM_TAG', value: 'latest'), string(name: 'BRANCH_TO_BUILD', value: 'develop')]
        
    if(params.DRY_RUN == false)
    {
        echo 'TAGGING!'
        
        git(
            url: "https://github.com/Inmarsat-itcloudservices/ESB-Relationship-${projectName}.git",
            credentialsId: '08089f00-d3e2-4ff4-bff6-5e585c07984a',
            branch: "develop"
        )
        
        withCredentials([usernamePassword(credentialsId: '08089f00-d3e2-4ff4-bff6-5e585c07984a', passwordVariable: 'password', usernameVariable: 'username')]) {
            echo "tagging"
            sh("git tag DH17")
            echo "pushing"
            sh('git push https://${username}:${password}@github.com/Inmarsat-itcloudservices/ESB-Relationship-${projectName}.git DH17')
        }
    }
}

JUNKins it is!

For anyone looking for escaping \ (single backslash) you need to use Double backslash

@hammerdj1

sh "git push https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/${org}/${repo}.git ${tag}"

Works for me.
This should also work:

sh 'git push https://$GIT_USERNAME:$GIT_PASSWORD@github.com/$org/$repo.git $tag'

difference being ' is string literal vs " is a templated string, or GString

@hammerdj1 you need to escape the templating

  foo = 'faz' # local groovy variable
  env.bar = 'baz' # add to shell environment
  sh "echo ${foo} templates while \\${bar} is pulled from shell environment"

Returns faz templates while baz is pulled from shell environment escaping the dollar sign lets it get passed from groovy gstrings to the shell execution.

Thanks for this post! I was getting mad trying to escape double quotes in jenkinsfiles. I have an issue though : When I triple escape my double quotes, it randomly adds single quotes in the string which is between those escapes :
sh(script: 'ansible-playbook /etc/ansible/playbooks/myplaybook.yml -e\\\"${VAR_2} arg1=true arg2=false\\\"')
Jenkins reads it like this :
ansible-playbook /etc/ansible/playbooks/myplaybook.yml '-e"Name=hc-dps-test' arg1=true 'arg2=false"'
As you can see it added 3 single quotes in the sh command!!!

Does anybody have the solution? I tried everything I could think of with no luck...

In my case the default shell happened to be 'dash', so a saner solution was to just switch to bash:

sh "#!/bin/bash\n" + """echo "with 'quotes'!" """

dollar-slashy should work too, and no excaping needed

You can also switch Jenkins default shell to bash under Manage Jenkins > Configure System ... Shell and pointing it to /bin/bash.

I love this! Also, while I appreciate that it may be working "as expected", that is a bit besides the point. The point is that it's currently very difficult to wrap inline shell scripts in a Jenkinsfile. Whatever the reason, this should be enhanced to where the user can simply drop in a shell command, and have that command passed, completely intact, to the shell. I like the idea of being able to invoke any local executable, but can see how this could be considered a security risk...but post a warning and allow the admin to approve/decline it?

I was struggling with quoting some commands that needed to pass through both Jenkins' sh as well as ssh. Specifically we ended up with some shared memory segments getting left behind on machines that needed to get cleaned up. The command I wanted to run across these hosts was:

ssh [host] /usr/bin/ipcs -m | awk '$6==0 && $3=="jenkins" { print $2 }' | xargs -r -n1 ipcrm -m

After struggling for too long I resorted to a hack I've seen elsewhere: base64-encoding the quoted string and passing that through instead:

$ base64 -w0
/usr/bin/ipcs -m | awk '$6==0 && $3=="jenkins" { print $2 }' | xargs -r -n1 ipcrm -m
[ctrl-d]
L3Vzci9iaW4vaXBjcyAtbSB8IGF3ayAnJDY9PTAgJiYgJDM9PSJqZW5raW5zIiB7IHByaW50ICQyIH0nIHwgeGFyZ3MgLXIgLW4xIGlwY3JtIC1tCg==

Then my Jenkins sh command becomes:

sh "ssh $host 'echo L3Vzci9iaW4vaXBjcyAtbSB8IGF3ayAnJDY9PTAgJiYgJDM9PSJqZW5raW5zIiB7IHByaW50ICQyIH0nIHwgeGFyZ3MgLXIgLW4xIGlwY3JtIC1tCg== | base64 -d | bash'"

Admittedly it's a hack, and obviously important to include a comment on exactly what you're doing, but it's effective and should work with arbitrarily complex quoting without worrying about how Jenkins interprets your quoting.

@shuffman For that particular use case, I would probably use the ssh plugin. However, you can also put double-quoted strings inside of groovy vars and call them.

def foo="jenkins"
sh (script: /bash -c "ssh [host] /usr/bin/ipcs -m | awk '$6==0 && $3==$foo { print $2 }' | xargs -r -n1 ipcrm -m"/ returnStatus: true)

def foo = 'bar'
sh 'echo "${foo}"' // (returns nothing)

Isn't the above expected? Groovy doesn't interpolate variables in single-quotes, so shell would see ${foo} and since we are not setting any shell variable named foo, shell replaces it with empty string. You can see it in action if you change it like this: sh 'foo=bar; echo "${foo}"' as you get "bar" as output.

def command = $/echo \\\"${foo}\\\"/$
sh command // \"bar\"

I think this was split because, together it would generate unexpected char: '\' error. However, just put parentheses around it and it would work fine: sh($/echo \\\"${foo}\\\"/$)

Madness

I have seen some comments state that it's the way Groovy is doing the interpolation. However, one can test escapes using the online Groovy console. I ran into an issue where the Groovy console needed three \\ (escapes) but by the time it ran through the Jenkins pipeline I had to add another two for a total of five escapes. This is the only documentation I have seen on this subject matter. The Jenkins doc's show very misleading simple sh(script:) commands. I will refrain from bashing Jenkins here though. No pun intended.

How in the fuck do I execute a shell command that needs both a literal dollar sign and interpolated values?

I need Jenkins to execute this: sh "curl -X POST -u \$literalUserName:${ftpProfile.password} …...

But Jenkins refuses to honor the escaped dollar sign and tries to replace it with a variable named "literalUserName". Why is it this bad.

@JasonFreeberg are you trying to have it say $literalUserName with the dollar sign in the string? That is a Bash (or I guess with Jenkins, possibly Dash) thing. See the gist, a single backslash gets swallower. To feed $ to the shell you need to feed \$ to Jenkins. You can use the dollar slashy strings too where escaping the dollar sign is simply $$literalUserName.

@Faheetah -- Yes, I am trying to preserve the dollar sign in the string "$literalUserName". I need it preserved in the final command, but either Jenkins, Groovy, or Bash tries to replace the string with a variable or env variable. I have tried the following:

sh "curl -X POST -u \$freebergtomcat:${ftpProfile.password} --data-binary @target/calculator-1.0.war https:/\${webAppName}.scm.azurewebsites.net/api/wardeploy"
sh 'curl -X POST -u \$freebergtomcat'+":${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy"
sh 'curl -X POST -u $freebergtomcat'+":${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy"
sh "curl -X POST -u \\$freebergtomcat:${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy"
sh '''
  curl -X POST -u \\\\\$freebergtomcat:${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy
'''

And this gives me a "bad substitution" error:

 sh 'curl -X POST -u \\$freebergtomcat:${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy'

Got it

So I concatenated a literal string and a GString. Still need to double escape the dollar in the literal string. I guess once for Groovy and a second time for bash? Man this sucks!

sh 'curl -X POST -u \\$freebergtomcat'+":${ftpProfile.password} --data-binary @target/calculator-1.0.war https://${webAppName}.scm.azurewebsites.net/api/wardeploy"

While you may be tempted to use groovy variables in scripts, I personally think that it's wrong way to go. Use of groovy vars makes script both hard to read and test. Getting input validation right is just a nightmare. And if you use secrets, you are leaking them on disk.

For all these reasons I decided to write a simple groovy fragment that tries to solve all these issues. It passes arguments to a delegate script as environment variables. Environment variables are then passed to actual script as command line arguments. Rather than explaining how it works, you can check it from: https://github.com/samlink-devops/jenkins-pipeline-examples/blob/master/vars/safeSh.groovy .

Using the script-generator as suggested by a few people helped me. The sh command seems to behave better when you specify the script parameter:

sh script: 'aws ses send-email --from no-reply@domain.com --to ops@domain.com --subject \'Drift Analysis\' --text \'Analysis complete!\''

Without it I seemed to be getting the errors suggested above, loosing the quotes in the command, when I tried 2 or 6 backslashes the parameters would be split and so on.

I think the biggest problem I see when it comes to bash and groovy is people escape the bash but then forget they have to escape the groovy.

Let us look at what we can't use in bash courtesy of stackexchange

String badText = '''
! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.
'''

println badText
~
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 23: unexpected char: '\' @ line 23, column 1.
   \ — shell syntax.
   ^

1 error

If you try to run this string its not going to work. You can't even print this string to console. It doesn't matter what type of groovy string you use to wrap it. ' ''' " """ / just not going to work.

That's because it has an illegal groovy character in it. So let's escape the \ as \\. Great, we can print this string to the console but can it run in a sh() step?

String badText = '''
! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.
'''

println badText

sh("echo '${badText}'")
~
[Pipeline] echo

! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.

[Pipeline] sh
+ echo 
! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
 — shell syntax.

! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
 — shell syntax.
tmp/durable-eb6837bd/script.sh: Syntax error: "(" unexpected (expecting ")")
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

So as expected this string is not safe for bash. Notice we get pretty far down the list? That's because when I echo it in the sh() step I wrap it in a single quote to turn of bash interpolation.

So how will we escape all these nasty things? A regex would work. Something like this [^-A-Za-z0-9._\s] Then for each problem string like ( we could wrap it in single quotes and escape it like this '\('. Also you dont have to wrap it in single quotes. I need to because in my use case I'm in a set of single quotes already.

Something like this would work String goodText = badText.replaceAll('[^-A-Za-z0-9._\s]'){ "'\${it}'" }

But if you run than you get errors? Because we havent escaped the groovy!

String goodText = badText.replaceAll('[^-A-Za-z0-9._\\s]'){ "'\\${it}'" } this will work. Lets see it all together.

String badText = '''
! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.
'''
String goodText = badText.replaceAll('[^-A-Za-z0-9._\\s]'){ "'\\${it}'" }

println "Escaped text: \n${goodText}"

sh("echo '${goodText}'")
~
[Pipeline] echo
Escaped text: 

'\!' '\—' history expansion.
'\"' '\—' shell syntax.
'\#' '\—' comment start when preceded by whitespace'\;' zsh wildcards.
'\$' '\—' shell syntax.
'\&' '\—' shell syntax.
'\'' '\—' shell syntax.
'\(' '\—' even in the middle of a word'\:' ksh extended globs '\('also available in bash and zsh'\)''\;' zsh wildcards.
'\)' '\('see '\(''\)'
'\*' '\—' sh wildcard.
'\,' '\—' only inside brace expansion.
'\;' '\—' shell syntax.
'\<' '\—' shell syntax.
'\=' '\—' in zsh'\,' when it'\''s at the beginning of a file name '\('filename expansion with PATH lookup'\)'.
'\>' '\—' shell syntax.
'\?' '\—' sh wildcard.
'\[' '\—' sh wildcard.
'\\' '\—' shell syntax.
'\]' '\—' you may get away with leaving it unquoted.
'\^' '\—' history expansion'\;' zsh wildcard.
'\`' '\—' shell syntax.
'\{' '\—' brace expansion.
'\|' '\—' shell syntax.
'\}' '\—' needs to be escaped in zsh'\,' other shells are more lenient when there'\''s no matching opening brace.
'\~' '\—' home directory expansion when it'\''s at the beginning of a file name'\;' zsh wildcard'\;' always safe when it'\''s the last character.
A few more characters can require special handling sometimes'\:'

- isn'\''t special for the shell'\,' but when it'\''s at the beginning of a command argument'\,' it indicates an option. It can'\''t be protected with quotes since the special handling is in the command'\,' not in the shell. To protect a file name that begins with -'\,' you can put .'\/' before it '\—' this way it'\''s still the same file'\,' but the argument doesn'\''t begin with - anymore.
. isn'\''t special in itself'\,' but dot files are excluded from '\*' globs by default.
'\:' isn'\''t special for the shell'\,' but some commands parse it specially'\,' e.g. to indicate a remote file '\('hostname'\:'filename'\)'. Consult the documentation of the command to see how to cope with file nams containing colons.

[Pipeline] sh
+ echo 
! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.


! — history expansion.
" — shell syntax.
# — comment start when preceded by whitespace; zsh wildcards.
$ — shell syntax.
& — shell syntax.
' — shell syntax.
( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
) (see ()
* — sh wildcard.
, — only inside brace expansion.
; — shell syntax.
< — shell syntax.
= — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
> — shell syntax.
? — sh wildcard.
[ — sh wildcard.
\ — shell syntax.
] — you may get away with leaving it unquoted.
^ — history expansion; zsh wildcard.
` — shell syntax.
{ — brace expansion.
| — shell syntax.
} — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.
A few more characters can require special handling sometimes:

- isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
. isn't special in itself, but dot files are excluded from * globs by default.
: isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.

Hopefully this will help you next time you need to escape something in a bash script/command running on Jenkins.

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!
something like
sh "find . -name *.xml -exec ls \;"

is becoming...
find . -name '*.xml' -exec ls '';

I am having exactly same problem, did you find a solution?

@jjaaskel for non quotes you can probably get away with two backslashes only because quotes get swallowed specifically due to multiple passes of interpreting quotes. If it doesn't work, try either the six backslashes or dollar slashy strings. Generally two should be sufficient since it will just escape the backslash to a backslash.

@jjaaskel for non quotes you can probably get away with two backslashes only because quotes get swallowed specifically due to multiple passes of interpreting quotes. If it doesn't work, try either the six backslashes or dollar slashy strings. Generally two should be sufficient since it will just escape the backslash to a backslash.

Sorry, I was not maybe accurate enough with my question. So to be exact, my issue is that while I have like this in Jenkins pipeline:
sh "find */config.xml"

It will be interpreted in shell like like this:
find '*/config.xml'

Which does not then work in shell as expected. Any proposal how it should be solved (meaning single quotes would not exist in shell command)?

@jjaaskel that sounds more like something to report up as an issue to Jenkins. In my gist I only address how to preserve quotes as I went through trying to figure it out and wanted to save people the hassle. I will say that if you are going strictly by the output in the console, it is wrong. I don't know why, but I have went down rabbit holes before trying to figure out quotes when they should be working, so might be worth verifying that is actually the case. What you put does look exactly what I fought with before, I just don't remember the specifics anymore.

Praise the heavens you named this Jenkinsfile idiosynchrasies instead of something like Jenksinfile behaviour with quotes, it makes so much easy to google it whenever I'm in doubt.... "Jenkins idiosomething"...

Anyways thank you for this amazing gist

@Faheetah - Thanks so much for this excellent page - really helpful

@jjaaskel - I was having the same issue as you when I came across this page.
I'm no Jenkins expert and I don't know how others will have their pipelines defined, but I was referencing a groovy script within my Jenkinsfile with a load command. In my groovy script, the command sh "mv foo/*/* foo/" was being interpreted in shell like yours as mv 'foo/*/*' foo/ - which caused my pipeline to fail.
I eventually got around this by moving the command into the Jenkinsfile rather than the groovy script. In the Jenkinsfile, the phantom single quotes aren't being added.
I also played around with putting it into an executable script.sh file and executing that from within the groovy, before settling on the above.

Hope this helps