Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Jenkinsfile idiosynchrasies with escaping and quotes
node {
echo 'No quotes, pipeline command in single quotes'
sh 'echo $BUILD_NUMBER'
echo 'Double quotes are silently dropped'
sh 'echo "$BUILD_NUMBER"'
echo 'Even escaped with a single backslash they are dropped'
sh 'echo \"$BUILD_NUMBER\"'
echo 'Using two backslashes, the quotes are preserved'
sh 'echo \\"$BUILD_NUMBER\\"'
echo 'Using three backslashes still results in preserving the single quotes'
sh 'echo \\\"$BUILD_NUMBER\\\"'
echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
def foo = 'bar'
sh 'echo "${foo}"'
echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
sh "echo \"${foo}\""
echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
sh 'echo \\\\\\"' + foo + '\\\\\\"'
echo 'We still needed all of that escaping and mixing concatenation is hideous!'
echo 'There must be a better way, enter dollar slashy strings (actual term)'
def command = $/echo \\\"${foo}\\\"/$
sh command
echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
sh vash
echo 'It still requires escaping the escape but that is just bash being bash at that point'
echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}
@flyinprogrammer

This comment has been minimized.

Show comment
Hide comment
@flyinprogrammer

flyinprogrammer May 21, 2016

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

@magnayn

This comment has been minimized.

Show comment
Hide comment
@magnayn

magnayn Aug 8, 2016

I think "idiosynchrasies" is being overly polite!

magnayn commented Aug 8, 2016

I think "idiosynchrasies" is being overly polite!

@dgladyshev

This comment has been minimized.

Show comment
Hide comment
@dgladyshev

dgladyshev Sep 3, 2016

thanks, you saved my day

thanks, you saved my day

@gsaslis

This comment has been minimized.

Show comment
Hide comment
@gsaslis

gsaslis Sep 27, 2016

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

gsaslis commented Sep 27, 2016

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

@dramirezp

This comment has been minimized.

Show comment
Hide comment
@dramirezp

dramirezp Dec 22, 2016

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

@gigi-at-zymergen

This comment has been minimized.

Show comment
Hide comment
@gigi-at-zymergen

gigi-at-zymergen Mar 6, 2017

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

@laapsaap

This comment has been minimized.

Show comment
Hide comment
@laapsaap

laapsaap Mar 18, 2017

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

@leti-ulloa

This comment has been minimized.

Show comment
Hide comment
@leti-ulloa

leti-ulloa May 4, 2017

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

@andresvia

This comment has been minimized.

Show comment
Hide comment
@andresvia

andresvia May 12, 2017

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

@andresvia

This comment has been minimized.

Show comment
Hide comment
@andresvia

andresvia May 12, 2017

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

@nathanwelch

This comment has been minimized.

Show comment
Hide comment
@nathanwelch

nathanwelch May 17, 2017

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName
@jamesj2

This comment has been minimized.

Show comment
Hide comment
@jamesj2

jamesj2 Aug 4, 2017

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

jamesj2 commented Aug 4, 2017

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

@james-gu

This comment has been minimized.

Show comment
Hide comment
@james-gu

james-gu Aug 11, 2017

Thank you for creating this, the \\\\\\\ saved me here!

Thank you for creating this, the \\\\\\\ saved me here!

@upjersdave

This comment has been minimized.

Show comment
Hide comment
@upjersdave

upjersdave Aug 17, 2017

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

@dolfinsbizou

This comment has been minimized.

Show comment
Hide comment
@dolfinsbizou

dolfinsbizou Sep 20, 2017

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

@jurgenweber

This comment has been minimized.

Show comment
Hide comment
@jurgenweber

jurgenweber Oct 25, 2017

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

@Faheetah

This comment has been minimized.

Show comment
Hide comment
@Faheetah

Faheetah Nov 14, 2017

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

Owner

Faheetah commented Nov 14, 2017

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

@mdebord1

This comment has been minimized.

Show comment
Hide comment
@mdebord1

mdebord1 Dec 5, 2017

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

mdebord1 commented Dec 5, 2017

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

@Ibmurai

This comment has been minimized.

Show comment
Hide comment
@Ibmurai

Ibmurai Mar 22, 2018

Even with this guide I just spent 3+ hours with this :@

Ibmurai commented Mar 22, 2018

Even with this guide I just spent 3+ hours with this :@

@laapsaap

This comment has been minimized.

Show comment
Hide comment
@laapsaap

laapsaap Apr 5, 2018

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

laapsaap commented Apr 5, 2018

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

@tbye-pfp

This comment has been minimized.

Show comment
Hide comment
@tbye-pfp

tbye-pfp Apr 11, 2018

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

@aaronsilverman

This comment has been minimized.

Show comment
Hide comment
@aaronsilverman

aaronsilverman Apr 14, 2018

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

aaronsilverman commented Apr 14, 2018

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

@abakhru

This comment has been minimized.

Show comment
Hide comment
@abakhru

abakhru Apr 16, 2018

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

abakhru commented Apr 16, 2018

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 16, 2018

@laapsaap Lol!, Indeed i had one..

ghost commented May 16, 2018

@laapsaap Lol!, Indeed i had one..

@Faheetah

This comment has been minimized.

Show comment
Hide comment
@Faheetah

Faheetah May 21, 2018

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

Owner

Faheetah commented May 21, 2018

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

@sleungcy

This comment has been minimized.

Show comment
Hide comment
@sleungcy

sleungcy Jun 14, 2018

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';

sleungcy commented Jun 14, 2018

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';
@tbye-pfp

This comment has been minimized.

Show comment
Hide comment
@tbye-pfp

tbye-pfp Jun 19, 2018

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

@Alexhha

This comment has been minimized.

Show comment
Hide comment
@Alexhha

Alexhha Jun 20, 2018

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }

Alexhha commented Jun 20, 2018

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment