Jenkinsfile idiosynchrasies with escaping and quotes

Jenkinsfile.groovy

node {
    echo 'Results included as an inline comment exactly how they are returned as of Jenkins 2.121, with $BUILD_NUMBER = 1'
    echo 'No quotes, pipeline command in single quotes'
    sh 'echo $BUILD_NUMBER' // 1
    echo 'Double quotes are silently dropped'
    sh 'echo "$BUILD_NUMBER"' // 1
    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"' // 1
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"' // "1"
    echo 'Using three backslashes still results in only preserving the quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"' // "1"
    echo 'To end up with \" use \\\\\\" (yes, six backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
    echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
    def foo = 'bar'
    sh 'echo "${foo}"' // (returns nothing)
    echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
    sh "echo \"${foo}\"" // bar
    echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
    sh 'echo \\\\\\"' + foo + '\\\\\\"' // \"bar\"
    echo 'We still needed all of that escaping and mixing concatenation is hideous!'
    echo 'There must be a better way, enter dollar slashy strings (actual term)'
    def command = $/echo \\\"${foo}\\\"/$
    sh command // \"bar\"
    echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
    def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
    sh vash // \"3\" bar
    echo 'It still requires escaping the escape but that is just bash being bash at that point'
    echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}

Migrate to GitHub Actions people.

…

On Mon, 7 Nov 2022, 9:34 pm Burhan Madraswala, ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ @sourabhgupta385 <https://github.com/sourabhgupta385> Can u please help here in this code for Jenkins pipeline I am trying to execute this WP CLI command via bash script in Jenkins job where in i am escaping all the special characters but still while executing the Jenkins job it throws me an error. Command: wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\"#https?://(www\\\\.)?(|${WP_ENGINE}. wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}. wpenginepowered.com/wp-$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([ <http://wpenginepowered.com/wp-$4%5Cn#https?://$%7BWP_ENGINE_CDN%7D(.wpengine%7C-wpengine).netdna-(ssl%7Ccdn).com/([>^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/$3.pdf\n#\ <http://wpengine.com/$3.pdf%5Cn#%5C>"/wp-content/themes/browserstack/img# => \"https://${WP_ENGINE}. wpenginepowered.com/wp-content/themes/browserstack/img\ <http://wpenginepowered.com/wp-content/themes/browserstack/img%5C>";' WHERE 1=1 AND option_name='regex_html_post_process'" The above code gets successfully executed on terminal but on Jenkins it renders like this and throws an error /home/wpe-user/sites/bsengineering/command.sh: line 24: syntax error near unexpected token (' /home/wpe-user/sites/bsengineering/command.sh: line 24: wp db query "UPDATE wp_options SET option_value='s:434:"#https?://(www\.)?(| bsengineering.wpengine.com|3hmev4rik691dp80q3b650u1.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://bsengineering.wpenginepowered.com/wp-' [image: Screenshot 2022-11-07 at 4 02 29 PM] <https://user-images.githubusercontent.com/17512774/200289134-c3ea11e2-8f0a-4e2b-a8cc-5f7e79a1cf80.png> [image: Screenshot 2022-11-07 at 4 02 14 PM] <https://user-images.githubusercontent.com/17512774/200289143-d338d9b9-2c86-4f29-bb21-1dccfa2f12fb.png> Attached SS for reference from Jenkins job — Reply to this email directly, view it on GitHub <https://gist.github.com/e11bd0315c34ed32e681616e41279ef4#gistcomment-4360877> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA42VRP2AWSQ6MIBZXMHDDWHDLKBBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVAZTKNJUGAZDKOFHORZGSZ3HMVZKMY3SMVQXIZI> . You are receiving this email because you commented on a thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

@burhanuddin7 Looking at the command line executed, I think the issue comes from the second double quote not being escaped properly, it ends the query wp db query "UPDATE wp_options SET option_value='s:434:"

Beware you need two backslashes instead of one in groovy : https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4#file-jenkinsfile-groovy-L9
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https? etc

Hope that helps, I might be missing a lot of context here.

@michaelPf85 If I add two backslashes instead of one in bash script from which i am executing the code on local then it throws me the error
Command:
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https?://(www\\\\.)?(|${WP_ENGINE}.wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}.wpenginepowered.com/wp-$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/$3.pdf\n#\\"/wp-content/themes/browserstack/img# => \\"https://${WP_ENGINE}.wpenginepowered.com/wp-content/themes/browserstack/img\\";' WHERE 1=1 AND option_name='regex_html_post_process'"

will this code work in jenkins job for the same? or am i missing out something

Migrate to GitHub Actions people.
…

No thanks.

Migrate to GitHub Actions people.

I would if I could but we can't use GH or GHE for our codebase due to compliance reasons so we are stuck with Jenkins. GH actions would make my life so easy, but we have a good bit of customizatiom in Jenkins too. I would recommend people use actions if possible, or another tool, but not everyone can.

@burhanuddin7 If you are indeed writing your shell script in a Jenkinsfile, you simply CAN NOT test your command locally.

• A command that runs locally, with backslashes, won't work in a Jenkinsfile.
• A command that runs in a Jenkinsfile, with backslashes as documented in this gist, won't work locally.

So my recommandation (doubling the backslash) should only be tested in the Jenkins job.

As another solution, to be able to test your command locally, you could extract your command in a separate .sh file, but it adds other challenges, it's not as simple to deploy and run.

@michaelPf85 Thanks for the explanation!!
But if i test the code in Jenkins job will this code with proper backslashes and escape characters work without giving error in pipeline job
Command:
wp db query "UPDATE wp_options SET option_value='s:${total_word_count}:\\"#https?://(www\\\\.)?(|${WP_ENGINE}.wpengine.com|${WP_ENGINE_CDN}.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://${WP_ENGINE}.wpenginepowered.com/wp-\\\$4\n#https?://${WP_ENGINE_CDN}(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://${WP_ENGINE}.wpengine.com/\\\$3.pdf\n#\\"/wp-content/themes/bs/img# => \\"https://${WP_ENGINE}.wpenginepowered.com/wp-content/themes/bs/img\\";' WHERE 1=1 AND option_name='regex_html_post_process'"

FYI: I have also escape '$' with triple backslashes to print $ within the string, will the above code work correctly without syntax error?
While running the query in the job it wont throw this error now correct?
Error: syntax error near unexpected token ('`

@burhanuddin7 Honestly I am not able to know if the code will work correctly. So the only 'advice' I can give you is to test at each step, looking at the command-line generated by Jenkins in the log, and find by yourself for each error where the escaping is wrong.

I know it's a pain but if anyone finds a better solution, I'd love to hear about it, it would save us all a lot of time !

@michaelPf85 the query above is working fine in Jenkins file but i have one small issue, in my rule_one and rule_two variable at he end i need a line break for which i have added '\n' but while compiling and giving final output it does not line break instead it gives me an empty space as output.
Can you please help me with this ? How do i line break my variable in the query so that it gives proper output

Code:::
**rule_one**='\\\\\\"#https?://(www\\\\.)?(|'${WP_ENGINE}'.wpengine.com|'${WP_ENGINE_CDN}'.wpengine.netdna-(ssl|cdn).com)/wp-(content|includes)# => https://'${WP_ENGINE}'.wpenginepowered.com/wp-\\\$4\n' **rule_two**='#https?://'${WP_ENGINE_CDN}'(.wpengine|-wpengine).netdna-(ssl|cdn).com/([^\\\\.]*).pdf# => https://'${WP_ENGINE}'.wpengine.com/\\\$3.pdf\n'

Query::
wp db query "UPDATE wp_options SET option_value='s:120:${rule_one}${rule_two}xyz' WHERE 1=1 AND option_name='regex_html_post_process'"

Can u please help here, I need line break after rule_one and rule_two variables
How to do that in groovy jenkins job

@michaelPf85 can you please help me out in my above query posted

You just need to escape the $:

pipeline {
	environment {
		TEST = "It's easy!"
	}

	stages {
		stage('Test') {
			steps {
				sh 'echo "\$BUILD_NUMBER:   \$TEST"'
			}
		}
	}
}

Works as expected:

[Pipeline] sh
+ echo '1:   It'\''s easy!'
1:   It's easy!