Instantly share code, notes, and snippets.

Embed
What would you like to do?
Jenkinsfile idiosynchrasies with escaping and quotes
node {
echo 'Results included as an inline comment exactly how they are returned as of Jenkins 2.121, with $BUILD_NUMBER = 1'
echo 'No quotes, pipeline command in single quotes'
sh 'echo $BUILD_NUMBER' // 1
echo 'Double quotes are silently dropped'
sh 'echo "$BUILD_NUMBER"' // 1
echo 'Even escaped with a single backslash they are dropped'
sh 'echo \"$BUILD_NUMBER\"' // 1
echo 'Using two backslashes, the quotes are preserved'
sh 'echo \\"$BUILD_NUMBER\\"' // "1"
echo 'Using three backslashes still results in only preserving the quotes'
sh 'echo \\\"$BUILD_NUMBER\\\"' // "1"
echo 'To end up with \" use \\\\\\" (yes, six backslashes)'
sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
def foo = 'bar'
sh 'echo "${foo}"' // (returns nothing)
echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
sh "echo \"${foo}\"" // bar
echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
sh 'echo \\\\\\"' + foo + '\\\\\\"' // \"bar\"
echo 'We still needed all of that escaping and mixing concatenation is hideous!'
echo 'There must be a better way, enter dollar slashy strings (actual term)'
def command = $/echo \\\"${foo}\\\"/$
sh command // \"bar\"
echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
sh vash // \"3\" bar
echo 'It still requires escaping the escape but that is just bash being bash at that point'
echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}
@flyinprogrammer

This comment has been minimized.

flyinprogrammer commented May 21, 2016

for build number you could also just do this i believe? ${env.BUILD_NUMBER}

@magnayn

This comment has been minimized.

magnayn commented Aug 8, 2016

I think "idiosynchrasies" is being overly polite!

@dgladyshev

This comment has been minimized.

dgladyshev commented Sep 3, 2016

thanks, you saved my day

@gsaslis

This comment has been minimized.

gsaslis commented Sep 27, 2016

@nowayride I was struggling with interpolation in multiline strings this morning...

FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation

so interpolating in ''' did not work, whereas interpolating in """ worked just fine.

@dramirezp

This comment has been minimized.

dramirezp commented Dec 22, 2016

Hi I'm having some issues trying to do this :
sh """ sed -i -e 's/%1//g' \${variable}/foo.txt """
Looks like the "%" is getting some issues in the jenkins file, any idea? also I tried with $1 but it is not working

@gigi-at-zymergen

This comment has been minimized.

gigi-at-zymergen commented Mar 6, 2017

Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend.

@laapsaap

This comment has been minimized.

laapsaap commented Mar 18, 2017

Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code.

@leti-ulloa

This comment has been minimized.

leti-ulloa commented May 4, 2017

Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement ..

@andresvia

This comment has been minimized.

andresvia commented May 12, 2017

I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args.

Like:

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

or a new pipeline command to avoid overloading "sh"

sh(script: "echo", args: ["hello", "world", env.MY_ENV, my_other_def])

On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.

["/bin/echo", "foo", "bar"].execute()

Is not common to have groovy's execute() allowed on Jenkins sandboxed environment, maybe for good reasons...

@andresvia

This comment has been minimized.

andresvia commented May 12, 2017

I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231

@nathanwelch

This comment has been minimized.

nathanwelch commented May 17, 2017

Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:

    def imageTag = 'feature/someBranchName'
    def command = $/echo ${imageTag} | sed 's/\//\-/'/$
    
    imageTag = sh(returnStdout: true, script: command).trim()
    sh("echo ${imageTag}")
    // outputs feature-someBranchName
@jamesj2

This comment has been minimized.

jamesj2 commented Aug 4, 2017

I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script.

@james-gu

This comment has been minimized.

james-gu commented Aug 11, 2017

Thank you for creating this, the \\\\\\\ saved me here!

@upjersdave

This comment has been minimized.

upjersdave commented Aug 17, 2017

Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important.

@gjorando

This comment has been minimized.

gjorando commented Sep 20, 2017

I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?)

@jurgenweber

This comment has been minimized.

jurgenweber commented Oct 25, 2017

'$/curl' -i ........ /$'
/tmp/jenkins8907199902083656367.sh: line 21: $/curl: No such file or directory

doesn't work?

@Faheetah

This comment has been minimized.

Owner

Faheetah commented Nov 14, 2017

@jurgenweber not sure but is your line actually

'$/curl' -i ........ /$'

With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try

$/curl -i http://example.com//$

Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so.

@ghost

This comment has been minimized.

ghost commented Dec 5, 2017

The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together.

@Ibmurai

This comment has been minimized.

Ibmurai commented Mar 22, 2018

Even with this guide I just spent 3+ hours with this :@

@laapsaap

This comment has been minimized.

laapsaap commented Apr 5, 2018

Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming.

We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place.

@tbye-pfp

This comment has been minimized.

tbye-pfp commented Apr 11, 2018

Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes".
Also, I'm not sure what you are trying to say in line 2: 'No quotes in single backticks'. For one, the character (') is a single quote not a single backtick. Secondly, I don't know what you mean by the "No quotes" part of that sentence. Do you mean you don't need to use quotes within an echo within an sh?
When writing something like this, where the subject is already confusing, I think it's very important to use the correct names for the characters so you don't confuse people more:
' single quote
" double quote
` backtick
\ backslash

@aaronsilverman

This comment has been minimized.

aaronsilverman commented Apr 14, 2018

I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors.

@abakhru

This comment has been minimized.

abakhru commented Apr 16, 2018

Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc.

@ghost

This comment has been minimized.

ghost commented May 16, 2018

@laapsaap Lol!, Indeed i had one..

@Faheetah

This comment has been minimized.

Owner

Faheetah commented May 21, 2018

@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script.

@aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too.

@abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review.

@sleungcy

This comment has been minimized.

sleungcy commented Jun 14, 2018

how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash!

something like

sh "find . -name *.xml -exec ls \\;"

is becoming...

find . -name '*.xml' -exec ls '\';
@tbye-pfp

This comment has been minimized.

tbye-pfp commented Jun 19, 2018

Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4.

@Alexhha

This comment has been minimized.

Alexhha commented Jun 20, 2018

Maybe would be useful

bash script

$ cat test.sh
#!/bin/bash

args='-Dexec.args='"'"'${project.groupId}'"'"''
echo "${args}"

$ ./test.sh
-Dexec.args='${project.groupId}'

Pipeline script

node('master'){
   stage('STAGE1'){
      cmd_args='-Dexec.args=\\\'\\${project.groupId}\\\''
      sh "echo $cmd_args"
   }
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/bash-var-expansion
[Pipeline] {
[Pipeline] stage
[Pipeline] { (STAGE1)
[Pipeline] sh
[bash-var-expansion] Running shell script
+ echo -Dexec.args='${project.groupId}'
-Dexec.args='${project.groupId}'
[Pipeline] }
@gzzo

This comment has been minimized.

gzzo commented Jun 28, 2018

For anyone that ends up here, the Snippet Generator is your best friend.

@michaelPf85

This comment has been minimized.

michaelPf85 commented Jul 10, 2018

@Faheetah Nice gist ! Could you add %, that you have to double ? Like sh "echo '1 % 2'" doesn't display the %, you have to double it : "1 %% 2"

@videte47

This comment has been minimized.

videte47 commented Aug 3, 2018

seeing this gist made me realized I am not alone on this pain "yay"

@CH3T

This comment has been minimized.

CH3T commented Aug 22, 2018

I am stuck with this for two friggin days :-/ .

sh 'docker exec tnd-chrome bash -c sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\'--tags @Regression\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'

this causes sudo to throw usage usage: sudo -h | -K | -k | -V etc...

		sh "docker exec tnd-chrome bash -c \\'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\\\'--tags @Regression\\\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify\\'"

this causes mvn: -c: line 0: unexpected EOF while looking for matching `''
mvn: -c: line 1: syntax error: unexpected end of file

:/ Any advice greatly appreciated

@CH3T

This comment has been minimized.

CH3T commented Aug 22, 2018

		sh "docker exec tnd-chrome bash -c 'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\\\'--tags @Regression\\\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'"

unexpected EOF while looking for matching `''

@CH3T

This comment has been minimized.

CH3T commented Aug 22, 2018

Tears of joy :') :')

This worked!!!

sh "docker exec tnd-chrome bash -c 'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\"--tags @Regression\" -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'"

@mcandre

This comment has been minimized.

mcandre commented Sep 7, 2018

The situation is much worse than this. Any space-separated arguments inside of quotes sent to Jenkins' awful sh task are broken up, single-quoted, and recombined against the mangled outer quoting. Christ, just pass the data through as it was input!

@rudolfwalter

This comment has been minimized.

rudolfwalter commented Sep 8, 2018

All of this works as expected, people.

The quotes are not dropped by anything in Jenkins. They are interpreted by your shell. Try running some of those commands in bash manually, and you'll see the same behaviour. That is,
echo a
and
echo "a"
will both output the single character a. When you see the shell command being run in the log (the lines that begin with a +), it is not Jenkins showing it to you before sending it to the shell; rather, it is the shell itself echoing it back, as started with the -x (aka -o xtrace) flag.

The backslashes on the other hand are being processed by both Jenkins and the shell. That is, when you write \\\\\\\" as you show above, Jenkins eats one set of escapes, and sends \\\" to the shell; then the shell does one more round, sending a literal \" to echo.

Note, however, that you almost never want this, most executables do not expect literal quotes in their parameters – those are for the shell.

You can easily test that things are working as intended with a shell script that lists its parameters line-by-line:

#!/bin/bash
echo "Got $# args:"
while [ $# -gt 0 ]; do
    echo "$1"
    shift
done
echo

Save this as something like printargs.sh and then replace every echo in your example with it.

Oh, and replace $BUILD_NUMBER with some variable that has spaces in its value, eg:

node {
    withEnv(['FOO=a b c']) {
        sh 'printargs.sh $FOO'
        sh 'printargs.sh "$FOO"'
    }
}
@CH-DanReif

This comment has been minimized.

CH-DanReif commented Sep 17, 2018

Quoth the @rudolfwalter:

All of this works as expected, people.

The quotes are not dropped by anything in Jenkins. They are interpreted by your shell.

Exactly right. Taking it line-by-line, starting at https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L4-L5 :

   echo 'Double quotes are silently dropped'
   sh 'echo "$BUILD_NUMBER"'

I'll use $USER as my example variable, since it's (pretty much) always present in Bash, whereas $BUILD_NUMBER only exists inside Jenkins runs. So, in Bash, this is equivalent to typing:

echo "$USER"

Which, as expected, generates just bmdan.

Next up, https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L6-L9 :

    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"'
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"'

What we're seeing here is Groovy's escaping. The rules are as follows:

  1. After sh, look for the next string literal opener.
  2. Load the string (honoring escapes), until you hit the corresponding (unescaped) closer for your opener.
  3. Send the string to the system(3) or execve(2) call.

In the first case, Groovy helpfully honors a meaningless escape (\" inside of single quotes is just a literal ", same as it would have been without the \). Thus, after step 2, we're sending echo "$BUILD_NUMBER" to the shell, same as the previous two lines. In the second case, Groovy's escaping drops the \\ down to just a \, and sends that through, so the final command sent to the shell is echo \"$BUILD_NUMBER\". Try that as echo \"$USER\" in your shell and you'll see the same output: quotes around the var.

Moving on to https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L10-L11 :

    echo 'Using three backslashes still results in preserving the single quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"'

Three backslashes followed by a " now makes perfect sense, because Groovy sees this as two escapes: the first, an escaped backslash. The second, a (needlessly-) escaped ". So when this goes into the shell, it's echo \"$BUILD_NUMBER\", indistinguishable from the two slashes example above it.

Finally, we have https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L12-L13 :

    echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'

First off, no, you have six backslashes here. Which is exactly how many it should take. The final string sent to the shell is echo \\\"$BUILD_NUMBER\\\": an escaped \ and an escaped ", concatenated, on each side of the variable. It takes six backslashes to make sure that three backslashes end up at the shell once Groovy's done with it. Mind you, doing this is almost certainly not what you wanted to do; this prints out literal quotes and backslashes. If you're doing that in preparation for loading this into something else, you want to escape the contents of the variable, too, or something like echo \\\\\\"$PATH\\\\\\" is going to go straight to hell if there's an embedded " or \ inside $PATH.

Bonus: if you do happen to use seven backslashes, though, you get the exact same output. Think through why: any odd number of backslashes is simply going to hand some number of backslashes to the shell, and then escape the character following the last backslash in Groovy. Thus, final backslash will never show up by the time the shell gets it, the same as the 1- and 3-backslash formulations, above.

@Faheetah

This comment has been minimized.

Owner

Faheetah commented Sep 28, 2018

@michaelPf85 added, thanks!

I'm surprised this gist is still so active, good info everyone. I don't always see replies so best bet for more prompt feedback would probably be to post on SO or something, but I'll try to keep this updated with suggestions as I see them.

Edit: um, this doesn't look to be the case. When I use sh 'echo something with %%' it comes out with both of them.

@Faheetah

This comment has been minimized.

Owner

Faheetah commented Sep 28, 2018

@CH-DanReif I'm going to test run this and include the results as of Jenkins 2.121 since it should help a bit, and let me error check this (might have been updates since I wrote it). Looks like it is indeed six backslashes and I've updated the echo to note that, although seven technically works because of the aforementioned triple backslash. And yes there are some better ways of handling that but there are use cases for passing a literal "$var" to the shell. This gist doesn't reflect best bash practices but rather how to get around how clunky backslashes are going through several layers of escaping. It may not be a var but a string with a dollar sign in it too.

@agray

This comment has been minimized.

agray commented Oct 31, 2018

So how would I escape this dos batch command for a Jenkinsfile bat command?

%OC_EXE% -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test DNC.XTest\DNC.XTest.csproj --test-adapter-path:. --logger:xunit;LogFilePath=..\xunit-test-results.xml" -output:"%RAWOUTPUT%" -filter:+[*]* -oldStyle

slashy string don't seem to work, or I haven't nutted it yet.

In my Jenkinsfile I have this:
bat "${env.OC_EXE} -register:user -target:${env.DOTNET_EXE} -targetargs:'test --test-adapter-path:. --logger:xunit;LogFilePath=../${env.UT_RESULTS}' -output:../${env.RAWOUTPUT} -filter:+[*]* -oldStyle"

Which I can see in the console produces this:
G:\Jenkins\workspace\POC\Pipeline\DNCTest\DNC.XTest>C:\Users\xxx/.nuget/packages/opencover/4.6.519/tools/OpenCover.Console.exe -register:user -target:C:/PROGRA~1/dotnet/dotnet.exe -targetargs:'test --test-adapter-path:. --logger:xunit;LogFilePath=../xunit-test-results.xml' -output:../raw_coverage.xml -filter:+[*]* -oldStyle

But is throwing this error:
Incorrect Arguments: The argument '--test-adapter-path' is not recognised

How the heck do I correctly wrap -targetargs correctly so groovy will accept it and OpenCover knows to just pass --test-adapter-path onto dotnet.exe??

@Faheetah

This comment has been minimized.

Owner

Faheetah commented Nov 5, 2018

@agray this comes more down to how does Jenkins handle and escape spaces. This is a blight on any code trying to wrap shell commands, you come across it in any language and framework. Usually it's done with lists separating args so you get something like:

doShell(['ls', '-ahl', 'some directory'])

Which gets interpreted the same as running:

$ ls -ahl some\ directory

Jenkins, from all I've seen, has decided to use only strings as shell input. If you are dealing with too many types of things to escape, it might be easier to use an explicit script then call that. I haven't used Jenkins with Windows scripting but with Linux I believe you have to escape each quote or it disappears, so you would have to do something akin to:

sh 'foo --bar=\\\"something with spaces\\\"'
@somenick

This comment has been minimized.

somenick commented Nov 22, 2018

In my case the default shell happened to be 'dash', so a saner solution was to just switch to bash:

sh "#!/bin/bash\n" + """echo "with 'quotes'!" """

dollar-slashy should work too, and no excaping needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment