Jenkinsfile idiosynchrasies with escaping and quotes

Jenkinsfile.groovy

node {
    echo 'No quotes, pipeline command in single quotes'
    sh 'echo $BUILD_NUMBER'
    echo 'Double quotes are silently dropped'
    sh 'echo "$BUILD_NUMBER"'
    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"'
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"'
    echo 'Using three backslashes still results in preserving the single quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"'
    echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
    echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
    def foo = 'bar'
    sh 'echo "${foo}"'
    echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
    sh "echo \"${foo}\""
    echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
    sh 'echo \\\\\\"' + foo + '\\\\\\"'
    echo 'We still needed all of that escaping and mixing concatenation is hideous!'
    echo 'There must be a better way, enter dollar slashy strings (actual term)'
    def command = $/echo \\\"${foo}\\\"/$
    sh command
    echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
    def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
    sh vash
    echo 'It still requires escaping the escape but that is just bash being bash at that point'
    echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

I think "idiosynchrasies" is being overly polite!

I think "idiosynchrasies" is being overly polite!

thanks, you saved my day

thanks, you saved my day

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

Thank you for creating this, the \\\\\\\ saved me here!

Thank you for creating this, the \\\\\\\ saved me here!

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

Even with this guide I just spent 3+ hours with this :@

Even with this guide I just spent 3+ hours with this :@

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

@laapsaap Lol!, Indeed i had one..

@laapsaap Lol!, Indeed i had one..

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }

[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }

[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }

For anyone that ends up here, the Snippet Generator is your best friend.

For anyone that ends up here, the Snippet Generator is your best friend.

@Faheetah Nice gist ! Could you add %, that you have to double ? Like sh "echo '1 % 2'" doesn't display the %, you have to double it : "1 %% 2"

@Faheetah Nice gist ! Could you add %, that you have to double ? Like sh "echo '1 % 2'" doesn't display the %, you have to double it : "1 %% 2"

seeing this gist made me realized I am not alone on this pain "yay"

seeing this gist made me realized I am not alone on this pain "yay"