Skip to content

Instantly share code, notes, and snippets.

@Faheetah
Last active March 6, 2024 18:14
Star You must be signed in to star a gist
Save Faheetah/e11bd0315c34ed32e681616e41279ef4 to your computer and use it in GitHub Desktop.
Jenkinsfile idiosynchrasies with escaping and quotes
node {
echo 'Results included as an inline comment exactly how they are returned as of Jenkins 2.121, with $BUILD_NUMBER = 1'
echo 'No quotes, pipeline command in single quotes'
sh 'echo $BUILD_NUMBER' // 1
echo 'Double quotes are silently dropped'
sh 'echo "$BUILD_NUMBER"' // 1
echo 'Even escaped with a single backslash they are dropped'
sh 'echo \"$BUILD_NUMBER\"' // 1
echo 'Using two backslashes, the quotes are preserved'
sh 'echo \\"$BUILD_NUMBER\\"' // "1"
echo 'Using three backslashes still results in only preserving the quotes'
sh 'echo \\\"$BUILD_NUMBER\\\"' // "1"
echo 'To end up with \" use \\\\\\" (yes, six backslashes)'
sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
def foo = 'bar'
sh 'echo "${foo}"' // (returns nothing)
echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
sh "echo \"${foo}\"" // bar
echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
sh 'echo \\\\\\"' + foo + '\\\\\\"' // \"bar\"
echo 'We still needed all of that escaping and mixing concatenation is hideous!'
echo 'There must be a better way, enter dollar slashy strings (actual term)'
def command = $/echo \\\"${foo}\\\"/$
sh command // \"bar\"
echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
sh vash // \"3\" bar
echo 'It still requires escaping the escape but that is just bash being bash at that point'
echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}
@Rajdeep-689
Copy link

Rajdeep-689 commented Nov 3, 2020

@fortil I know I'm bad at the quotes that's why asked in the forum, so that someone can share some knowledge or one sample... So next time onwards I might not ask the same question trust me.. I checked the example and tried all possible ways (triple slash \\ or single quote ' or slash" slash")what I understood, but it's not happening. I tried base64 string encoding as well, but in that the Jenkins params are not resolving. Yeah if we take any hardcoded folder location or file name in the powershell command then that's a really good solution. Anyways thanks for your time, if I will be able to resolve it then definitely share the sample with you..

@shadycuz
Copy link

shadycuz commented Nov 3, 2020

@Rajdeep-689

I dont have access to a windows node to test this, but in general you can think of your command like this...

"power shell command "inner command "quoted parameter"""

Try not too nest so many of the same quotes together.

'power shell command "inner command \'quoted parameter\'"'

Something sorta like this except I don't remember how to escape on windows 😊

Stack overflow devops would be a decent place to ask this question.

Also if Physical_FolderLoc and Stord_Process_Name are not groovy variables then they need escaped as well.

@Rajdeep-689
Copy link

@shadycuz @fortil @haridsv @shadycuz @agray

I got the exact issue why the command was not escaping. There is a stand alone $ symbol in the powershell command which is also a system character for groovy, so that need to be escaped as well.(The $ character which is after ForEach-Object {). It's now prefixed with \, so escaped.

Now the complete command is :
bat "powershell -Command \"(Get-Content ${Physical_FolderLoc}\\${Stord_Process_Name}.txt) | ForEach-Object { \$_ -replace [RegEx]::Escape('E:\\config'), 'I:\\config' } | Set-Content ${Physical_FolderLoc}\\${Stord_Process_Name}.txt\" "

Thanks a lot for the suggestion and thanks everyone for your time.

@carpiupin
Copy link

I'm trying to execute the following maven command with jenkins sh:
mvn release:perform -Darguments="-Dmaven.javadoc.skip=true -DskipTests"

But jenkins always introduce random single quotes and it doesn't work...

Ex1: sh 'mvn release:perform -Darguments="-Dmaven.javadoc.skip=true -DskipTests"' // Run: mvn release:perform '-Darguments=-Dmaven.javadoc.skip=true -DskipTests'
Ex2: sh 'mvn release:perform -Darguments=\\"-Dmaven.javadoc.skip=true -DskipTests\\"' // Run: mvn release:perform '-Darguments="-Dmaven.javadoc.skip=true' '-DskipTests"'
Ex3: sh 'mvn release:perform -Darguments=\\\"-Dmaven.javadoc.skip=true -DskipTests\\\"' // Run: mvn release:perform '-Darguments="-Dmaven.javadoc.skip=true' '-DskipTests"'
...

How can I do this with jenkins sh or why are single quotes in the resulting command?

@Faheetah
Copy link
Author

@carpiupin

Ignore the output of the quoted, Jenkins does something wrong when displaying them on the logs. What you see is probably not what Jenkins is running. The sh step gets pretty unhappy with spaces in commands.

It looks like it should work as is, but maybe try dollar slashy strings? You might need to escape the space. I would also check if the maven plugin can do what you are trying to accomplish, I try to avoid shell commands any chance I get for this reason.

@JuaanO
Copy link

JuaanO commented Apr 14, 2021

How to suppose escape of this:

sh 'mvn test -Dcucumber.options="-t @"${params.Primera}""'

@shadycuz
Copy link

@JuaanO I'm not familiar with mvn's arguments but I suspect this will work.

sh "mvn test -Dcucumber.options='-t @${params.Primera}'"

if that doesn't work because you really need a set of quotes after the @ try:

sh """mvn test -Dcucumber.options='-t @"${params.Primera}"'"""

@MarinaGhe
Copy link

MarinaGhe commented Apr 14, 2021

Hello there, I'm trying to set a pipeline with Jenkins and given the fact that I'm new at this I have a big problem with interpolating a custom param variable into a shell command. I also have defined in my Jenkins configuration different deploy environments and in the deployment step if I do something like:

sh """ ${DEPLOY_TO_DEV} """
it works as expected and if I try to do something like:

environment { DEPLOY = "DEPLOY_TO_${params.ENV}" }
steps { sh "echo $DEPLOY" sh "${DEPLOY}" }

I get an error saying that the script was not found or is not executing anything, although it prints DEPLOY_TO_DEV. I've tried some suggestions found on google but without success, I can't figure out what I'm missing.
Any help is appreciated! Thanks

@shadycuz
Copy link

@MarinaGhe

A better place to get help is DevOps StackExhage or the Jenkins Gitter.

I'm not sure why your first example would work or not work because I would need to see the rest of your job.

I can however explain the second example.

// if the parameter ENV == 'STAGE'
environment {
    DEPLOY = "DEPLOY_TO_${params.ENV}"
}
// Then this would create an environment variable DEPLOY
// whose value is `DEPLOY_TO_STAGE
// this get passed directly to the shell as is and it works because
// you previously created an environment variable called DEPLOY
sh "echo $DEPLOY"
// This does not get passed directly to the shell.
// Groovy interpolates DEPLOY but I didn't see you create a variable called DEPLOY
// So this should error for variable not found OR maybe it get turned into `null` and passed like that to sh
// this explains your error about command not found
sh "${DEPLOY}"

It seems like you are using declarative pipelines and I don't know much about that, but in scripted pipeline I would use:

String command = "DEPLOY_TO_${params.ENV}"

sh(command)

I wouldn't mess with creating an environment variable unless the script it's self needed it.

@MarinaGhe
Copy link

@shadycuz

Thanks a lot for your response. The first command works because in my Jenkins configuration I have defined multiple global variables each associated with a deploy command and that's how I access it in my Jenkinsfile. Given the fact that I have multiple deploy environments I wanted to use the custom params value to execute a different deploy command depending on what param is selected. I was suspecting the behaviour you described but I wasn't sure. I haven't found the solution yet but I'll keep try.

@shadycuz
Copy link

@MarinaGhe

So you have global variables that are like DEPLOY_TO_DEV and DEPLOY_TO_STAGE?

You want users to enter a parameter called ENV when they run the job and then using that parameters value, you want the contents of the matching global variable?

as an example

String DEPLOY_TO_DEV = 'some_command --some ARGS` // DEPLOY_TO_DEV has the "dev" deploy command

String ENV = 'DEV' // User entered "DEV" for the ENV parameter

if (ENV == 'DEV") {
    sh(DEPLOY_TO_DEV)
}

If this example above is the correct logic then you can do something like this:

steps {
    script {
        if (params.ENV == 'DEV' ) {
            sh "${DEPLOY_TO_DEV}"
        }
        if (params.ENV == 'STAGE' ) {
            sh "${DEPLOY_TO_STAGE}"
        }
    }
}

I think as you get more advanced with jenkins you might want to stop using declarative and move to scripted. This excample in scripted:

Map commands = [
    'DEV' : 'command --arg',
    'STAGE': 'command --arg'
]

sh(commands[params.ENV])

Where commands is a dictionary, like in python and the params.ENV is used as the key.

@JuaanO
Copy link

JuaanO commented Apr 14, 2021

@shadycuz Thanks a lot for you response. Yes, works with:
sh """mvn test -Dcucumber.options='-t @"${params.Primera}"'"""

However now response come with doble quote, how to scape to @"Primera" ?

Running runner.RunCukesTest None of the features at [src/test/java/features] matched the filters: [@"Primera"]

@JuaanO
Copy link

JuaanO commented Apr 14, 2021

@shadycuz Thanks a lot for you response, Yeah works removing two quotes.

sh """mvn test -Dcucumber.options='-t @${params.Primera}'"""

Thanks @shadycuz

@MarinaGhe
Copy link

@shadycuz

That's exactly what I'm doing already :D (I should have provided the whole script, sorry) I was trying to find out if there is an alternative to the multiple if's and trying to dynamically create the deploy command. But I will look into the scripted pipeline also, seems to be more powerful. Thanks a lot for your time, you rock! Have a great day!

@celesteking
Copy link

Jenkins is a piece of garbage. A simple rename xx yy *.log turns to hell.

@sourabhgupta385
Copy link

sourabhgupta385 commented Aug 8, 2021

I want to execute below command in Jenkins
trivy image --format template --template "@contrib/html.tpl" -o trivy-report.html --input helloworld_${BUILD_NUMBER}.tar

In above command note double quotes with @

In Jenkinsfile, I have given instruction as :

sh script: 'trivy --cache-dir /tmp/trivy image --format template --template \"@contrib/html.tpl\" -o trivy-report.html --input helloworld_${BUILD_NUMBER}.tar'

but in jenkins logs I see:

[Pipeline] sh

  • trivy --cache-dir /tmp/trivy image --format template --template @contrib/html.tpl -o trivy-report.html --input helloworld__44.tar

i.e. double quotes is omitted. For the command double quotes and @ needs to be adjacent for it to run properly. Can anyone please help me with this? I've tried a lot of options but nothing works.

@haridsv
Copy link

haridsv commented Aug 8, 2021

@sourabhgupta385 You don't need backslashes for the double-quotes to be seen by shell, i.e., the below would make shell see the double-quotes just fine:

sh script: 'trivy --cache-dir /tmp/trivy image --format template --template "@contrib/html.tpl" -o trivy-report.html --input helloworld_${BUILD_NUMBER}.tar'

What you are seeing in the console is the exact command that is being executed, not the command that is seen by shell. You can test this out by using a standalone script, something like this:

$ cat /tmp/t.sh
set -x
echo "@contrib/html.tpl"
$ bash /tmp/t.sh
+ echo @contrib/html.tpl
@contrib/html.tpl

As you can see above, though the script has double-quotes, what shell shows on the console doesn't have it. The double-quotes are only useful to prevent some processing from occurring and they get stripped before the actual command gets executed.

I recommend explaining the problem that you are trying to solve so that someone could provide a better solution that what you are doing.

@sourabhgupta385
Copy link

@sourabhgupta385 You don't need backslashes for the double-quotes to be seen by shell, i.e., the below would make shell see the double-quotes just fine:

sh script: 'trivy --cache-dir /tmp/trivy image --format template --template "@contrib/html.tpl" -o trivy-report.html --input helloworld_${BUILD_NUMBER}.tar'

What you are seeing in the console is the exact command that is being executed, not the command that is seen by shell. You can test this out by using a standalone script, something like this:

$ cat /tmp/t.sh
set -x
echo "@contrib/html.tpl"
$ bash /tmp/t.sh
+ echo @contrib/html.tpl
@contrib/html.tpl

As you can see above, though the script has double-quotes, what shell shows on the console doesn't have it. The double-quotes are only useful to prevent some processing from occurring and they get stripped before the actual command gets executed.

I recommend explaining the problem that you are trying to solve so that someone could provide a better solution that what you are doing.

Hi @haridsv
I executed :
image

But in Jenkins the command executed is:

image

Since the argument was not in double quotes, the command failed. When I executed it in local with double quotes, everything worked fine.
Therefore, I either need double or single quotes around @contrib/html.tpl

@Faheetah
Copy link
Author

Faheetah commented Aug 8, 2021

The reason it looks like the quotes aren't there is because the Jenkins console will further mangle the command and make it look like the wrong command ran. I've had cases of using quotes and spaces where something like foo bar "baz faz" in the console looked like foo "bar "baz" faz" or something (I forget exactly but it looked awful). What you see in the console can't be trusted. And sadly in that specific case, none of us could figure out how to get Jenkins to play nice with the quotes around the args. You might try using two backslashes before the quote, Jenkins might be swallowing them up.

@haridsv
Copy link

haridsv commented Aug 8, 2021

@sourabhgupta385 The @ symbol is not special in most shells, so there are essentially no special chars in the entire string @contrib/html.tpl and so the command should in theory work without any sort of quotes around it (though having double quotes won't harm it). If it is not working for you, it is most likely due to the workspace missing that file, or not in the correct working directory at the time the command is run. Does the filename actually start with the @ symbol or is it a special character for trivy?

@sourabhgupta385
Copy link

The directory contrib does not start with @. It's a special syntax for trivy to get the report generated in html from the template stored in contrib directory.

My workspace and everything is fine.. somewhere I think @ is treated as special character in Jenkins due to which combination of quote and @ is ignoring quotes. If I put space in between of double quote and @, then those quotes are not ignored while seeing in Jenkins console logs but that way I do not get desired output also. Report is not generated in template format. Rather the report contains the string contrib/html.tpl

@scottmelhop
Copy link

After hours of searching, this saved me. Thankyou!

@pyeprog
Copy link

pyeprog commented May 12, 2022

thank you for the harsh truth!

@luke-hill
Copy link

luke-hill commented Aug 9, 2022

It would be good to add examples where you mix and match params and env variables.

Single quotes = ❌ - Params don't work in single quotes
Double quotes = ❕ You're using double quotes with Env variables. Please don't.

Now This "does work" but it's MANKY

sh 'ENV_KEY1=$ENV_KEY1 ENV_KEY2=direct_value ' + "PARAM_ONE=${params.PARAM_ONE} ./shell_script"

@ntwrkguru
Copy link

It would be good to add examples where you mix and match params and env variables.

Single quotes = ❌ - Params don't work in single quotes Double quotes = ❕ You're using double quotes with Env variables. Please don't.

Now This "does work" but it's MANKY

sh 'ENV_KEY1=$ENV_KEY1 ENV_KEY2=direct_value ' + "PARAM_ONE=${params.PARAM_ONE} ./shell_script"

For this, I wrap it in an env block instead of passing them inline.

@burhanuddin7
Copy link

burhanuddin7 commented Nov 7, 2022

@sourabhgupta385 Can u please help here in this code for Jenkins pipeline
I am trying to execute this WP CLI command via bash script in Jenkins job where in i am escaping all the special characters but still while executing the Jenkins job it throws me an error.
Command:
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\"#https?://(www\\\\.)?(|${WP_ENGINE}.wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}.wpenginepowered.com/wp-$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/$3.pdf\n#\"/wp-content/themes/bs/img# => \"https://${WP_ENGINE}.wpenginepowered.com/wp-content/themes/bs/img\";' WHERE 1=1 AND option_name='regex_html_post_process'"

The above code gets successfully executed on terminal but on Jenkins it renders like this and throws an error
/home/wpe-user/sites/bsengineering/command.sh: line 24: syntax error near unexpected token ('
/home/wpe-user/sites/bsengineering/command.sh: line 24: wp db query "UPDATE wp_options SET option_value='s:434:"#https?://(www\.)?(|bsengineering.wpengine.com|3hmev4rik691dp80q3b650u1.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://bsengineering.wpenginepowered.com/wp-'

Attached SS for reference from Jenkins job

Screenshot 2022-11-07 at 4 02 14 PM
Screenshot 2022-11-07 at 4 02 29 PM

Can someone help here please

@agray
Copy link

agray commented Nov 7, 2022 via email

@michaelPf85
Copy link

@burhanuddin7 Looking at the command line executed, I think the issue comes from the second double quote not being escaped properly, it ends the query wp db query "UPDATE wp_options SET option_value='s:434:"

Beware you need two backslashes instead of one in groovy : https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L9
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https? etc

Hope that helps, I might be missing a lot of context here.

@burhanuddin7
Copy link

@michaelPf85 If I add two backslashes instead of one in bash script from which i am executing the code on local then it throws me the error
Command:
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https?://(www\\\\.)?(|${WP_ENGINE}.wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}.wpenginepowered.com/wp-$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/$3.pdf\n#\\"/wp-content/themes/browserstack/img# => \\"https://${WP_ENGINE}.wpenginepowered.com/wp-content/themes/browserstack/img\\";' WHERE 1=1 AND option_name='regex_html_post_process'"

will this code work in jenkins job for the same? or am i missing out something

@ntwrkguru
Copy link

Migrate to GitHub Actions people.

No thanks.

@Faheetah
Copy link
Author

Faheetah commented Nov 7, 2022

Migrate to GitHub Actions people.

I would if I could but we can't use GH or GHE for our codebase due to compliance reasons so we are stuck with Jenkins. GH actions would make my life so easy, but we have a good bit of customizatiom in Jenkins too. I would recommend people use actions if possible, or another tool, but not everyone can.

@michaelPf85
Copy link

@burhanuddin7 If you are indeed writing your shell script in a Jenkinsfile, you simply CAN NOT test your command locally.

  • A command that runs locally, with backslashes, won't work in a Jenkinsfile.
  • A command that runs in a Jenkinsfile, with backslashes as documented in this gist, won't work locally.

So my recommandation (doubling the backslash) should only be tested in the Jenkins job.

As another solution, to be able to test your command locally, you could extract your command in a separate .sh file, but it adds other challenges, it's not as simple to deploy and run.

@burhanuddin7
Copy link

burhanuddin7 commented Nov 7, 2022

@michaelPf85 Thanks for the explanation!!
But if i test the code in Jenkins job will this code with proper backslashes and escape characters work without giving error in pipeline job
Command:
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https?://(www\\\\.)?(|${WP_ENGINE}.wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}.wpenginepowered.com/wp-\\\$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/\\\$3.pdf\n#\\"/wp-content/themes/bs/img# => \\"https://${WP_ENGINE}.wpenginepowered.com/wp-content/themes/bs/img\\";' WHERE 1=1 AND option_name='regex_html_post_process'"

FYI: I have also escape '$' with triple backslashes to print $ within the string, will the above code work correctly without syntax error?
While running the query in the job it wont throw this error now correct?
Error: syntax error near unexpected token ('`

@michaelPf85
Copy link

@burhanuddin7 Honestly I am not able to know if the code will work correctly. So the only 'advice' I can give you is to test at each step, looking at the command-line generated by Jenkins in the log, and find by yourself for each error where the escaping is wrong.

I know it's a pain but if anyone finds a better solution, I'd love to hear about it, it would save us all a lot of time !

@burhanuddin7
Copy link

burhanuddin7 commented Nov 18, 2022

@michaelPf85 the query above is working fine in Jenkins file but i have one small issue, in my rule_one and rule_two variable at he end i need a line break for which i have added '\n' but while compiling and giving final output it does not line break instead it gives me an empty space as output.
Can you please help me with this ? How do i line break my variable in the query so that it gives proper output

Code:::
**rule_one**='\\\\\\"#https?://(www\\\\.)?(|'${WP_ENGINE}'.wpengine.com|'${WP_ENGINE_CDN}'.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://'${WP_ENGINE}'.wpenginepowered.com/wp-\\\$4\n' **rule_two**='#https?://'${WP_ENGINE_CDN}'(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://'${WP_ENGINE}'.wpengine.com/\\\$3.pdf\n'

Query::
wp db query "UPDATE wp_options SET option_value='s:120:${rule_one}${rule_two}xyz' WHERE 1=1 AND option_name='regex_html_post_process'"

Can u please help here, I need line break after rule_one and rule_two variables
How to do that in groovy jenkins job

@burhanuddin7
Copy link

@michaelPf85 can you please help me out in my above query posted

@vmiliantsei
Copy link

You just need to escape the $:

pipeline {
	environment {
		TEST = "It's easy!"
	}

	stages {
		stage('Test') {
			steps {
				sh 'echo "\$BUILD_NUMBER:   \$TEST"'
			}
		}
	}
}

Works as expected:

[Pipeline] sh
+ echo '1:   It'\''s easy!'
1:   It's easy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment