node { | |
echo 'Results included as an inline comment exactly how they are returned as of Jenkins 2.121, with $BUILD_NUMBER = 1' | |
echo 'No quotes, pipeline command in single quotes' | |
sh 'echo $BUILD_NUMBER' // 1 | |
echo 'Double quotes are silently dropped' | |
sh 'echo "$BUILD_NUMBER"' // 1 | |
echo 'Even escaped with a single backslash they are dropped' | |
sh 'echo \"$BUILD_NUMBER\"' // 1 | |
echo 'Using two backslashes, the quotes are preserved' | |
sh 'echo \\"$BUILD_NUMBER\\"' // "1" | |
echo 'Using three backslashes still results in only preserving the quotes' | |
sh 'echo \\\"$BUILD_NUMBER\\\"' // "1" | |
echo 'To end up with \" use \\\\\\" (yes, six backslashes)' | |
sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"' | |
echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings' | |
def foo = 'bar' | |
sh 'echo "${foo}"' // (returns nothing) | |
echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes' | |
sh "echo \"${foo}\"" // bar | |
echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly' | |
sh 'echo \\\\\\"' + foo + '\\\\\\"' // \"bar\" | |
echo 'We still needed all of that escaping and mixing concatenation is hideous!' | |
echo 'There must be a better way, enter dollar slashy strings (actual term)' | |
def command = $/echo \\\"${foo}\\\"/$ | |
sh command // \"bar\" | |
echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars' | |
def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$ | |
sh vash // \"3\" bar | |
echo 'It still requires escaping the escape but that is just bash being bash at that point' | |
echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine' | |
} |
This comment has been minimized.
This comment has been minimized.
I think "idiosynchrasies" is being overly polite! |
This comment has been minimized.
This comment has been minimized.
thanks, you saved my day |
This comment has been minimized.
This comment has been minimized.
@nowayride I was struggling with interpolation in multiline strings this morning... FWIW, my issue was that interpolation is NOT supported in single and triple single quoted strings: http://docs.groovy-lang.org/latest/html/documentation/#_string_interpolation so interpolating in |
This comment has been minimized.
This comment has been minimized.
Hi I'm having some issues trying to do this : |
This comment has been minimized.
This comment has been minimized.
Thank you for compiling this. Have been fighting some Jenkinsfile string handling oddities, and this helped me nail them down. dollar slashy strings are turning out to be my bestest friend. |
This comment has been minimized.
This comment has been minimized.
Man.. what a mess this is! Eventually when you get it working, its an eye sore in the code. |
This comment has been minimized.
This comment has been minimized.
Thank you for putting this list together.. spent most of the day trying to figure out a simple sh command with a sed replacement .. |
This comment has been minimized.
This comment has been minimized.
I believe Jenkins needs an args: parameter for the "sh" pipeline command, right now we have script:, returnStdout: and returnStatus: both useful to avoid messing with the filesystem. However, is clearly impossible or at least difficult to have clean (and secure) executions, with "sh", just my 2c, maybe Jenkins should do this automatically, but I'm not sure if I want that kind of magic, I rather depend on args. Like:
or a new pipeline command to avoid overloading "sh"
On both cases echo the program, not the shell built-in should be executed, Jenkins should look for the program on the systems $PATH/%PATH% but also an absolute path should be supported, like a regular groovy execute on a List.
Is not common to have groovy's |
This comment has been minimized.
This comment has been minimized.
I've converted the last comment into a Jenkins issue, let's see what happens. https://issues.jenkins-ci.org/browse/JENKINS-44231 |
This comment has been minimized.
This comment has been minimized.
Thank you for making this! To help others in the future, here's how you would do a pipe into sed. Don't need to add extra escapes:
|
This comment has been minimized.
This comment has been minimized.
I've found if you use the pipeline snippet generator it will escape your commands for you. Just select "sh: Shell Script" as your Sample Step and enter your command into the text area and click Generate Pipeline Script. |
This comment has been minimized.
This comment has been minimized.
Thank you for creating this, the |
This comment has been minimized.
This comment has been minimized.
Thank you! This just helped us with a very tricky problem. This is what happens when you let Java programmers write something important. |
This comment has been minimized.
This comment has been minimized.
I was about to burst in tears on my keyboard, then I found this. Thank you! (The fuck Jenkins, why would you do such things?) |
This comment has been minimized.
This comment has been minimized.
doesn't work? |
This comment has been minimized.
This comment has been minimized.
@jurgenweber not sure but is your line actually
With the quotes included? Looks like you have an extra quote in there, but even then the $/ replaces quotes so try
Not sure if you need to escape the forward slashes on the URL but likely will need http://example.com//$ if so. |
This comment has been minimized.
This comment has been minimized.
The time I waste on quoting in Jenkinsfiles is rediculous. Thank you for putting this together. |
This comment has been minimized.
This comment has been minimized.
Even with this guide I just spent 3+ hours with this :@ |
This comment has been minimized.
This comment has been minimized.
Seriously, groovy escaping in pipeline needs to be changed. It gets to a point that it is stupid and ridiculous to use. I used to love PIPELINE, but replacing bash scripts with groovy is one of the worst thing I ever done in my 20 years of programming. We have oneliners in bash that is just impossible to escape in groovy. So we are now forced to execute bash scripts in groovy. Which misses the whole point of using groovy the first place. |
This comment has been minimized.
This comment has been minimized.
Thanks for putting this together. I have a few suggestions. Line 12 needs an edit: "yes, seven backticks" should be "yes, seven backslashes". |
This comment has been minimized.
This comment has been minimized.
I'm not finding this to be accurate at all. I'm using a multibranch pipeline with Jenkins 2.116 and all the latest plugins as of April 13th 2018, and the dollar slashy solution still leads to errors. |
This comment has been minimized.
This comment has been minimized.
Easiest way is to use the Pipeline Snippet-Generator, it takes care of all backspacing etc. |
This comment has been minimized.
This comment has been minimized.
@laapsaap Lol!, Indeed i had one.. |
This comment has been minimized.
This comment has been minimized.
@tbye-pfp thanks, I was braindumping originally and that was incorrect. This was so long ago I don't remember what I intended since I don't use any backticks in the entire script. @aaronsilverman do you have an example snipped? This was a while back so they may have changed, but I would hope Jenkins wouldn't be that backwards incompatible. If that's the case I can update too. @abakhru yeah snippet generator helps a lot but there's sometimes pretty complex cases, and if it is automatically generating a ton of backslashes to do nested escapes it may end up with a lot less maintainable code than using dollar slashy syntax. Depends what you're doing but generators (not just in Jenkins) are typically just a good start and still benefit from human review. |
This comment has been minimized.
This comment has been minimized.
how do i use backslash in pipeline shell script? jenkins keeps trying to put single quotes around my backslash! something like
is becoming...
|
This comment has been minimized.
This comment has been minimized.
Thanks for making the edits! I think it's more clear, and now I understand what you are saying in lines 2 and 4. |
This comment has been minimized.
This comment has been minimized.
Maybe would be useful bash script
Pipeline script
|
This comment has been minimized.
This comment has been minimized.
For anyone that ends up here, the Snippet Generator is your best friend. |
This comment has been minimized.
This comment has been minimized.
@Faheetah Nice gist ! Could you add %, that you have to double ? Like sh "echo '1 % 2'" doesn't display the %, you have to double it : "1 %% 2" |
This comment has been minimized.
This comment has been minimized.
seeing this gist made me realized I am not alone on this pain "yay" |
This comment has been minimized.
This comment has been minimized.
I am stuck with this for two friggin days :-/ . sh 'docker exec tnd-chrome bash -c sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\'--tags @Regression\' -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify' this causes sudo to throw usage usage: sudo -h | -K | -k | -V etc...
this causes mvn: -c: line 0: unexpected EOF while looking for matching `'' :/ Any advice greatly appreciated |
This comment has been minimized.
This comment has been minimized.
unexpected EOF while looking for matching `'' |
This comment has been minimized.
This comment has been minimized.
Tears of joy :') :') This worked!!! sh "docker exec tnd-chrome bash -c 'sudo mvn -q -f tnd-test-automation/pom.xml --settings tnd-test-automation/maven-bdd-settings.xml -Dbrowser=chrome -Dweb-driver=chromedriverlinux.bin -Dlog-level=INFO -Dssh.username=${sshUsername} -Dssh.password=${sshPassword} -Denv=${tndEnv} -Dcucumber.options=\"--tags @Regression\" -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn clean verify'" |
This comment has been minimized.
This comment has been minimized.
The situation is much worse than this. Any space-separated arguments inside of quotes sent to Jenkins' awful |
This comment has been minimized.
This comment has been minimized.
All of this works as expected, people. The quotes are not dropped by anything in Jenkins. They are interpreted by your shell. Try running some of those commands in bash manually, and you'll see the same behaviour. That is, The backslashes on the other hand are being processed by both Jenkins and the shell. That is, when you write Note, however, that you almost never want this, most executables do not expect literal quotes in their parameters – those are for the shell. You can easily test that things are working as intended with a shell script that lists its parameters line-by-line:
Save this as something like Oh, and replace $BUILD_NUMBER with some variable that has spaces in its value, eg:
|
This comment has been minimized.
This comment has been minimized.
Quoth the @rudolfwalter:
Exactly right. Taking it line-by-line, starting at https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L4-L5 :
I'll use echo "$USER" Which, as expected, generates just Next up, https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L6-L9 :
What we're seeing here is Groovy's escaping. The rules are as follows:
In the first case, Groovy helpfully honors a meaningless escape ( Moving on to https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L10-L11 :
Three backslashes followed by a Finally, we have https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L12-L13 :
First off, no, you have six backslashes here. Which is exactly how many it should take. The final string sent to the shell is Bonus: if you do happen to use seven backslashes, though, you get the exact same output. Think through why: any odd number of backslashes is simply going to hand some number of backslashes to the shell, and then escape the character following the last backslash in Groovy. Thus, final backslash will never show up by the time the shell gets it, the same as the 1- and 3-backslash formulations, above. |
This comment has been minimized.
This comment has been minimized.
@michaelPf85 added, thanks! I'm surprised this gist is still so active, good info everyone. I don't always see replies so best bet for more prompt feedback would probably be to post on SO or something, but I'll try to keep this updated with suggestions as I see them. Edit: um, this doesn't look to be the case. When I use sh 'echo something with %%' it comes out with both of them. |
This comment has been minimized.
This comment has been minimized.
@CH-DanReif I'm going to test run this and include the results as of Jenkins 2.121 since it should help a bit, and let me error check this (might have been updates since I wrote it). Looks like it is indeed six backslashes and I've updated the echo to note that, although seven technically works because of the aforementioned triple backslash. And yes there are some better ways of handling that but there are use cases for passing a literal "$var" to the shell. This gist doesn't reflect best bash practices but rather how to get around how clunky backslashes are going through several layers of escaping. It may not be a var but a string with a dollar sign in it too. |
This comment has been minimized.
This comment has been minimized.
So how would I escape this dos batch command for a Jenkinsfile bat command?
slashy string don't seem to work, or I haven't nutted it yet. In my Jenkinsfile I have this: Which I can see in the console produces this: But is throwing this error: How the heck do I correctly wrap |
This comment has been minimized.
This comment has been minimized.
@agray this comes more down to how does Jenkins handle and escape spaces. This is a blight on any code trying to wrap shell commands, you come across it in any language and framework. Usually it's done with lists separating args so you get something like:
Which gets interpreted the same as running:
Jenkins, from all I've seen, has decided to use only strings as shell input. If you are dealing with too many types of things to escape, it might be easier to use an explicit script then call that. I haven't used Jenkins with Windows scripting but with Linux I believe you have to escape each quote or it disappears, so you would have to do something akin to:
|
This comment has been minimized.
This comment has been minimized.
In my case the default shell happened to be 'dash', so a saner solution was to just switch to bash:
dollar-slashy should work too, and no excaping needed |
This comment has been minimized.
This comment has been minimized.
Hello! Excuse me, when I use Blue Ocean, I want to extract the built-in variable $JOB_NAME of jenkins. For example, when extracting a-app job, I want job_name to be a-app, but the result of Blue Ocean is a-app/master, or a-app/dev, with an additional branch name. How should Jenkins file be written? |
This comment has been minimized.
This comment has been minimized.
First time I've ever written a Jenkins pipeline and it may be the last ;) This is driving me crazy. You can see below that I've created the variable projectName and used it successfully in many places. However when I use it in the 'git push' command it resolves to empty string. I've used ' quotes in this line. If I change it to " quotes, then the username & password variables don't seem to resolve properly. Any suggestions?
|
This comment has been minimized.
This comment has been minimized.
JUNKins it is! For anyone looking for escaping \ (single backslash) you need to use Double backslash |
This comment has been minimized.
This comment has been minimized.
Works for me.
difference being |
This comment has been minimized.
This comment has been minimized.
@hammerdj1 you need to escape the templating
Returns |
This comment has been minimized.
This comment has been minimized.
Thanks for this post! I was getting mad trying to escape double quotes in jenkinsfiles. I have an issue though : When I triple escape my double quotes, it randomly adds single quotes in the string which is between those escapes : Does anybody have the solution? I tried everything I could think of with no luck... |
This comment has been minimized.
This comment has been minimized.
You can also switch Jenkins default shell to bash under Manage Jenkins > Configure System ... Shell and pointing it to /bin/bash. |
This comment has been minimized.
This comment has been minimized.
I love this! Also, while I appreciate that it may be working "as expected", that is a bit besides the point. The point is that it's currently very difficult to wrap inline shell scripts in a Jenkinsfile. Whatever the reason, this should be enhanced to where the user can simply drop in a shell command, and have that command passed, completely intact, to the shell. I like the idea of being able to invoke any local executable, but can see how this could be considered a security risk...but post a warning and allow the admin to approve/decline it? |
This comment has been minimized.
This comment has been minimized.
I was struggling with quoting some commands that needed to pass through both Jenkins'
After struggling for too long I resorted to a hack I've seen elsewhere: base64-encoding the quoted string and passing that through instead:
Then my Jenkins
Admittedly it's a hack, and obviously important to include a comment on exactly what you're doing, but it's effective and should work with arbitrarily complex quoting without worrying about how Jenkins interprets your quoting. |
This comment has been minimized.
This comment has been minimized.
@shuffman For that particular use case, I would probably use the ssh plugin. However, you can also put double-quoted strings inside of groovy vars and call them. def foo="jenkins" |
This comment has been minimized.
This comment has been minimized.
Isn't the above expected? Groovy doesn't interpolate variables in single-quotes, so shell would see
I think this was split because, together it would generate |
This comment has been minimized.
This comment has been minimized.
Madness |
This comment has been minimized.
This comment has been minimized.
I have seen some comments state that it's the way Groovy is doing the interpolation. However, one can test escapes using the online Groovy console. I ran into an issue where the Groovy console needed three \\ (escapes) but by the time it ran through the Jenkins pipeline I had to add another two for a total of five escapes. This is the only documentation I have seen on this subject matter. The Jenkins doc's show very misleading simple sh(script:) commands. I will refrain from bashing Jenkins here though. No pun intended. |
This comment has been minimized.
This comment has been minimized.
How in the fuck do I execute a shell command that needs both a literal dollar sign and interpolated values? I need Jenkins to execute this: But Jenkins refuses to honor the escaped dollar sign and tries to replace it with a variable named "literalUserName". Why is it this bad. |
This comment has been minimized.
This comment has been minimized.
@JasonFreeberg are you trying to have it say $literalUserName with the dollar sign in the string? That is a Bash (or I guess with Jenkins, possibly Dash) thing. See the gist, a single backslash gets swallower. To feed $ to the shell you need to feed \$ to Jenkins. You can use the dollar slashy strings too where escaping the dollar sign is simply $$literalUserName. |
This comment has been minimized.
This comment has been minimized.
@Faheetah -- Yes, I am trying to preserve the dollar sign in the string "$literalUserName". I need it preserved in the final command, but either Jenkins, Groovy, or Bash tries to replace the string with a variable or env variable. I have tried the following:
And this gives me a "bad substitution" error:
Got itSo I concatenated a literal string and a GString. Still need to double escape the dollar in the literal string. I guess once for Groovy and a second time for bash? Man this sucks!
|
This comment has been minimized.
This comment has been minimized.
While you may be tempted to use groovy variables in scripts, I personally think that it's wrong way to go. Use of groovy vars makes script both hard to read and test. Getting input validation right is just a nightmare. And if you use secrets, you are leaking them on disk. For all these reasons I decided to write a simple groovy fragment that tries to solve all these issues. It passes arguments to a delegate script as environment variables. Environment variables are then passed to actual script as command line arguments. Rather than explaining how it works, you can check it from: https://github.com/samlink-devops/jenkins-pipeline-examples/blob/master/vars/safeSh.groovy . |
This comment has been minimized.
This comment has been minimized.
Using the script-generator as suggested by a few people helped me. The sh command seems to behave better when you specify the
Without it I seemed to be getting the errors suggested above, loosing the quotes in the command, when I tried 2 or 6 backslashes the parameters would be split and so on. |
This comment has been minimized.
This comment has been minimized.
I think the biggest problem I see when it comes to bash and groovy is people escape the bash but then forget they have to escape the groovy. Let us look at what we can't use in bash courtesy of stackexchange
If you try to run this string its not going to work. You can't even print this string to console. It doesn't matter what type of groovy string you use to wrap it. That's because it has an illegal groovy character in it. So let's escape the
So as expected this string is not safe for bash. Notice we get pretty far down the list? That's because when I echo it in the So how will we escape all these nasty things? A regex would work. Something like this Something like this would work But if you run than you get errors? Because we havent escaped the groovy!
Hopefully this will help you next time you need to escape something in a bash script/command running on Jenkins. |
This comment has been minimized.
This comment has been minimized.
I am having exactly same problem, did you find a solution? |
This comment has been minimized.
This comment has been minimized.
@jjaaskel for non quotes you can probably get away with two backslashes only because quotes get swallowed specifically due to multiple passes of interpreting quotes. If it doesn't work, try either the six backslashes or dollar slashy strings. Generally two should be sufficient since it will just escape the backslash to a backslash. |
This comment has been minimized.
This comment has been minimized.
Sorry, I was not maybe accurate enough with my question. So to be exact, my issue is that while I have like this in Jenkins pipeline: It will be interpreted in shell like like this: Which does not then work in shell as expected. Any proposal how it should be solved (meaning single quotes would not exist in shell command)? |
This comment has been minimized.
This comment has been minimized.
@jjaaskel that sounds more like something to report up as an issue to Jenkins. In my gist I only address how to preserve quotes as I went through trying to figure it out and wanted to save people the hassle. I will say that if you are going strictly by the output in the console, it is wrong. I don't know why, but I have went down rabbit holes before trying to figure out quotes when they should be working, so might be worth verifying that is actually the case. What you put does look exactly what I fought with before, I just don't remember the specifics anymore. |
This comment has been minimized.
This comment has been minimized.
Praise the heavens you named this Jenkinsfile idiosynchrasies instead of something like Jenksinfile behaviour with quotes, it makes so much easy to google it whenever I'm in doubt.... "Jenkins idiosomething"... Anyways thank you for this amazing gist |
This comment has been minimized.
This comment has been minimized.
@Faheetah - Thanks so much for this excellent page - really helpful @jjaaskel - I was having the same issue as you when I came across this page. Hope this helps |
This comment has been minimized.
This comment has been minimized.
This is amazing! Jenkins is not! |
This comment has been minimized.
This comment has been minimized.
Oh, don't be too harsh. It's still the best at what it does, even with it's nuances! |
This comment has been minimized.
This comment has been minimized.
Hello everyone, I have a question, how can I run sh script into the console script tool? |
This comment has been minimized.
This comment has been minimized.
same here, three years later :). This seems to apply for the secret text storage in the general settings. To have sed get a backslash in front of a forward slash, I needed to add the 6 backslashes in the secret string to have one in the eventual variable. Works though, wonder how long it would have taken to foind that without the post here. |
This comment has been minimized.
This comment has been minimized.
Thanks... :| |
This comment has been minimized.
This comment has been minimized.
Hey Guys. I have one powershell command inside that both Double quotes and single quotes are present and it's needed for sure in order to make the command execute successfully in dos prompt. I am not sure how to make it escape in pipeline script. bat "powershell -Command "(Get-Content "${Physical_FolderLoc}\${Stord_Process_Name}.txt") | ForEach-Object { $_ -replace [RegEx]::Escape('E:\config'), 'I:\config' } | Set-Content "${Physical_FolderLoc}\${Stord_Process_Name}.txt" " " In the above command you can see the second last " is ending quote for Get Content and last one is for bat command. |
This comment has been minimized.
This comment has been minimized.
Anyone who can address my concern please!!! I am seeking an expert advice. Please help. |
This comment has been minimized.
This comment has been minimized.
check the example and you will know how to do it, you're bad with the quotes, you have to scape them |
This comment has been minimized.
This comment has been minimized.
@fortil I know I'm bad at the quotes that's why asked in the forum, so that someone can share some knowledge or one sample... So next time onwards I might not ask the same question trust me.. I checked the example and tried all possible ways (triple slash \\ or single quote ' or slash" slash")what I understood, but it's not happening. I tried base64 string encoding as well, but in that the Jenkins params are not resolving. Yeah if we take any hardcoded folder location or file name in the powershell command then that's a really good solution. Anyways thanks for your time, if I will be able to resolve it then definitely share the sample with you.. |
This comment has been minimized.
This comment has been minimized.
I dont have access to a windows node to test this, but in general you can think of your command like this...
Try not too nest so many of the same quotes together.
Something sorta like this except I don't remember how to escape on windows Stack overflow devops would be a decent place to ask this question. Also if |
This comment has been minimized.
This comment has been minimized.
@shadycuz @fortil @haridsv @shadycuz @agray I got the exact issue why the command was not escaping. There is a stand alone $ symbol in the powershell command which is also a system character for groovy, so that need to be escaped as well.(The $ character which is after ForEach-Object {). It's now prefixed with \, so escaped. Now the complete command is : Thanks a lot for the suggestion and thanks everyone for your time. |
This comment has been minimized.
This comment has been minimized.
I'm trying to execute the following maven command with jenkins sh: But jenkins always introduce random single quotes and it doesn't work...
How can I do this with jenkins sh or why are single quotes in the resulting command? |
This comment has been minimized.
This comment has been minimized.
Ignore the output of the quoted, Jenkins does something wrong when displaying them on the logs. What you see is probably not what Jenkins is running. The sh step gets pretty unhappy with spaces in commands. It looks like it should work as is, but maybe try dollar slashy strings? You might need to escape the space. I would also check if the maven plugin can do what you are trying to accomplish, I try to avoid shell commands any chance I get for this reason. |
This comment has been minimized.
for build number you could also just do this i believe?
${env.BUILD_NUMBER}