Skip to content

Instantly share code, notes, and snippets.

@mavaddat

mavaddat/GPG_noninteractive_batch_sign_trust_and_send_gnupg_keys.md

Last active June 11, 2023 05:45
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mavaddat/a81627ec4680ef5d6c30a9ebdc412037 to your computer and use it in GitHub Desktop.
Save mavaddat/a81627ec4680ef5d6c30a9ebdc412037 to your computer and use it in GitHub Desktop.
Download ZIP
A script to search every contact in CSV file for publicly listed PGP or GPG key
Raw
contactsSearchGPG.ps1
# Downloaded MS Outlook-style CSV from Google Contacts
$contacts = Import-Csv -Path $env:TEMP\contacts.csv
# The following approach requires user input (select, confirm key to input)
foreach ($email in $contacts.'Email Address' | Where-Object { -not [string]::IsNullOrEmpty($_) }) {
gpg --auto-key-locate keyserver --search-keys $($email)
}
# The following does not require any user input (completely automated)
foreach ($email in $contacts.'Email Address' | Where-Object { -not [string]::IsNullOrEmpty($_) }) {
$keys = gpg --with-colons --auto-key-locate keyserver --locate-keys $email | Select-String -Pattern "(?<=fpr:+)[^:]+(?=:)"
foreach ($key in $keys.Matches.Value) {
gpg --auto-key-locate keyserver --recv-keys $key
}
}
Raw
example.csv
First Name Last Name Email Address
Hi Bye hi@bye.com
Yes No yes@no.com
Why Not why@not.com
Raw
getPubKeysFromCSV.sh
#!/bin/bash
# CSV of email address
csv=$1
# signature file
sig=$2
# file to verify
file=$3
# Get headers from CSV
headers=$(head -1 $csv)
# Find the column number of the email address
emailCol=$(echo $headers | tr ',' '\n' | grep -n "Email Address" | cut -d':' -f1)
# Content of the CSV at emailCol column, skipping the first line
emailAddrs=$(tail -n +2 $csv | cut -d',' -f$emailCol)
gpgListPatrn='(?<entropy>\d+)\s*bit\s*(?<algo>\S+)\s*key\s*(?<pubkeyid>[^,]+)'
# Loop through the array and get the public keys
for email in "${emailAddrs[@]}"
do
# Get the public key ids for the email address by named matching group 'pubkeyid' in the regex gpgListPatrn
pubkeyids=$(gpg --batch --keyserver hkp://keyserver.ubuntu.com --search-keys $email 2>&1 | grep -Po $gpgListPatrn | cut -d' ' -f5)
# For each public key id, get the public key
for pubkeyid in $pubkeyids
do
# Add the public key to the local keyring
recvr=$(gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys $pubkeyids 2>&1)
# If the public key is added, do some extra work with it
# [do stuff]
# For example, verify the signature $sig of $file
if [ -n "$recvr" ]; then
verify=$(gpg --batch --verify $sig $file 2>&1)
# If the signature is verified, announce it was verified
# else, print error not verified and exit
if [[ $verify =~ "^gpg: Good signature from" ]]; then
echo "$file was verified by $email using $pubkeyids"
else
printf '%s\n' "$file was unable to be verified" >&2
exit 1
fi
fi
done
done
Raw
GPG_noninteractive_batch_sign_trust_and_send_gnupg_keys.md

This is a text-only version of the following page on https://raymii.org:

Title : GPG noninteractive batch sign, trust and send gnupg keys Author : Remy van Elst Date : 01-06-2018 URL : https://raymii.org/s/articles/GPG_noninteractive_batch_sign_trust_and_send_gnupg_keys.html Format : Markdown/HTML

Recently a team I consult for started using a shared password manager, pass. It uses GPG keys and presents itself as the standard unix password manager, but in essence it's nothing more than a wrapper around GPG encrypted files. We all had to generate new keys since the team is new and we were not allowed to use existing keys. Using a new, empty keyring, I generated my key and imported their keys. I wanted to trust, sign and publish all keys to a keyserver, this article shows how to do that noninteractively in batch form. Saves me doing the same thing four times.

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

If you ever want to send me something encrypted, you can find my GPG and S/MIME keys here

I'm using the following GPG version on Ubuntu 18.04:

$ gpg --version
gpg (GnuPG) 2.2.4
libgcrypt 1.8.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/remy/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Do note that I started with an empty slate (new key, empty gnupg keyring) and with all people in the same room. Don't do this on your regular GnuPG keyring since then you might not want to trust and sign everyone's key. Normally you would just sign and publish the keys from people you actually verified the identity of, at a keysigning party for example.

Machine readable format

I started with an empty keyring, generated a new secret key and then imported the other keys from a folder (gpg --import *.asc). The next step is to trust these keys, sign them and upload them to a keyserver. I can do that by hand using the CLI, but that doesn't scale. This time it's three keys, the next time it will be a hundred. Let's find a way to automate that.

Using gpg --list-keys I can get a list of keys and their ID's:

$ gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   2  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   2  signed:   0  trust: 2-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2019-06-01
/home/remy/.gnupg/pubring.kbx
-----------------------------
pub   secp256k1 2018-06-01 [SC] [expires: 2019-06-01]
      BA3185A7E50F713280F4559AA2EB77DDEA029199
uid           [ultimate] R. van Elst <remy@example2.nl>
sub   secp256k1 2018-06-01 [E] [expires: 2019-06-01]

pub   rsa2048 2018-06-01 [SC] [expires: 2020-05-31]
      99A986134D54F69AB9BE0E3939110B67C1165E3F
uid           [ unknown] Christian <christian@example4.com
sub   rsa2048 2018-06-01 [E] [expires: 2020-05-31]

pub   rsa4096 2018-05-31 [SC]
      CF518D6D969A7448F7545A343A624724EAFF8D71
uid           [  full  ] Karel Jan van Dijk <karel@example1.org>
sub   rsa4096 2018-05-31 [E]

pub   rsa3072 2018-05-31 [SC] [expires: 2020-05-30]
      EC6D4F0A72117779D2D89C816F8ACF16C411318B
uid           [  full  ] Ron Duiker <ron@example1.org>
sub   rsa3072 2018-05-31 [E] [expires: 2020-05-30]

Parsing shell command output in general is a bad idea since that can change. GnuPG addresses this in the manual page specifically:

   --list-keys
   -k
   --list-public-keys
          List the specified keys.  If no keys are specified, then all keys from the configured public keyrings are listed.

          Never  use the output of this command in scripts or other programs.  The output is intended only for humans and its format is likely to change.  The --with-colons option emits the out-
          put in a stable, machine-parseable format, which is intended for use by scripts and other programs.

Thank you to the GnuPG developers for doing this, if all software would be so specific and clear that would save me a lot of time and effort.

The output --with-colons looks like this:

$ gpg --list-keys --with-colons
tru::1:1527866875:1559373557:3:1:5
pub:u:256:19:A2EB77DDEA029199:1527837557:1559373557::u:::scESC:::::secp256k1:::0:
fpr:::::::::BA3185A7E50F713280F4559AA2EB77DDEA029199:
uid:u::::1527837557::38561F394B0A364FE743B28BF03B3602E8F8D8E1::R. van Elst <remy@example2.nl>::::::::::0:
sub:u:256:18:770B8C8E75662ABE:1527837557:1559373557:::::e:::::secp256k1::
fpr:::::::::0E83BF12C86332243F76C8CC770B8C8E75662ABE:
pub:-:2048:1:39110B67C1165E3F:1527834492:1590906492::-:::scESC::::::23::0:
fpr:::::::::99A986134D54F69AB9BE0E3939110B67C1165E3F:
uid:-::::1527834492::6B405A1FBF0307CF08C19D6771B6F6472CB0E1FF::Christian <christian@example4.com::::::::::0:
sub:-:2048:1:4B7F18953DB2549A:1527834492:1590906492:::::e::::::23:
fpr:::::::::B5E0E76BD1A4338ADF67ACDC4B7F18953DB2549A:
pub:f:4096:1:3A624724EAFF8D71:1527781980:::-:::scESC::::::23::0:
fpr:::::::::CF518D6D969A7448F7545A343A624724EAFF8D71:
uid:f::::1527781980::F77BFCFB07762E657B2D8EB7BE6FFC348E6B9EC8::Karel Jan van Dijk <karel@example1.org>::::::::::0:
sub:f:4096:1:040F45BF5344AC48:1527781980::::::e::::::23:
fpr:::::::::D3F1C10C742313FFA998552C040F45BF5344AC48:
pub:f:3072:1:6F8ACF16C411318B:1527781700:1590853700::-:::scESC::::::23::0:
fpr:::::::::EC6D4F0A72117779D2D89C816F8ACF16C411318B:
uid:f::::1527781700::20DD8EAC4DE8E83947F95136BB03CD0E8ECE7D94::Ron Duiker <ron@example1.org>::::::::::0:
sub:f:3072:1:0492EAEFF2E0EA2F:1527781700:1590853700:::::e::::::23:
fpr:::::::::0B6AA791FB99C5FCE6701E260492EAEFF2E0EA2F:

This format is documented here. Reading that it seems I can safely search for 'fpr::::'. The change that someone names their key or comment that seems small to me.

Using awk I can get the fingerprints, which I need in the next commands to trust, sign and upload the keys:

$ gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $0}' 
fpr:::::::::4DDE73DB5030B53926813A502B6755BD1B7F88DC:
fpr:::::::::3AFDF7F4DA1B3A0C64671217CE2786EB97AC7685:
fpr:::::::::BA3185A7E50F713280F4559AA2EB77DDEA029199:
fpr:::::::::0E83BF12C86332243F76C8CC770B8C8E75662ABE:
fpr:::::::::99A986134D54F69AB9BE0E3939110B67C1165E3F:
fpr:::::::::B5E0E76BD1A4338ADF67ACDC4B7F18953DB2549A:
fpr:::::::::CF518D6D969A7448F7545A343A624724EAFF8D71:
fpr:::::::::D3F1C10C742313FFA998552C040F45BF5344AC48:
fpr:::::::::EC6D4F0A72117779D2D89C816F8ACF16C411318B:
fpr:::::::::0B6AA791FB99C5FCE6701E260492EAEFF2E0EA2F:

Printing just the fingerprints:

$ gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' 
4DDE73DB5030B53926813A502B6755BD1B7F88DC
3AFDF7F4DA1B3A0C64671217CE2786EB97AC7685
BA3185A7E50F713280F4559AA2EB77DDEA029199
0E83BF12C86332243F76C8CC770B8C8E75662ABE
99A986134D54F69AB9BE0E3939110B67C1165E3F
B5E0E76BD1A4338ADF67ACDC4B7F18953DB2549A
CF518D6D969A7448F7545A343A624724EAFF8D71
D3F1C10C742313FFA998552C040F45BF5344AC48
EC6D4F0A72117779D2D89C816F8ACF16C411318B
0B6AA791FB99C5FCE6701E260492EAEFF2E0EA2F

Trust the keys noninteractive in batch

Trusting a key involves going through a menu, setting a trust level and confirming that. Using the flag --command-fd 0 we instruct GnuPG to accept input from STDIN, thus allowing us to use a pipe with the correct input. The following command batch trusts all keys ultimately:

$ for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u); do  echo -e "5\ny\n" |  gpg --command-fd 0 --expert --edit-key $fpr trust; done

The output looks like this:

gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa3072/6F8ACF16C411318B
     created: 2018-05-31  expires: 2020-05-30  usage: SC  
     trust: ultimate      validity: ultimate
sub  rsa3072/0492EAEFF2E0EA2F
     created: 2018-05-31  expires: 2020-05-30  usage: E   
[ultimate] (1). Ron Duiker <ron@example1.org>

pub  rsa3072/6F8ACF16C411318B
     created: 2018-05-31  expires: 2020-05-30  usage: SC  
     trust: ultimate      validity: ultimate
sub  rsa3072/0492EAEFF2E0EA2F
     created: 2018-05-31  expires: 2020-05-30  usage: E   
[ultimate] (1). Ron Duiker <ron@example1.org>

Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu


pub  rsa3072/6F8ACF16C411318B
     created: 2018-05-31  expires: 2020-05-30  usage: SC  
     trust: ultimate      validity: ultimate
sub  rsa3072/0492EAEFF2E0EA2F
     created: 2018-05-31  expires: 2020-05-30  usage: E   
[ultimate] (1). Ron Duiker <>


pub  rsa3072/6F8ACF16C411318B
     created: 2018-05-31  expires: 2020-05-30  usage: SC  
     trust: ultimate      validity: ultimate
sub  rsa3072/0492EAEFF2E0EA2F
     created: 2018-05-31  expires: 2020-05-30  usage: E   
[ultimate] (1). Ron Duiker <ron@example1.org>

gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  secp256k1/A2EB77DDEA029199
     created: 2018-06-01  expires: 2019-06-01  usage: SC  
     trust: ultimate      validity: ultimate
ssb  secp256k1/770B8C8E75662ABE
     created: 2018-06-01  expires: 2019-06-01  usage: E   
[ultimate] (1). R. van Elst <remy@example2.nl>

sec  secp256k1/A2EB77DDEA029199
     created: 2018-06-01  expires: 2019-06-01  usage: SC  
     trust: ultimate      validity: ultimate
ssb  secp256k1/770B8C8E75662ABE
     created: 2018-06-01  expires: 2019-06-01  usage: E   
[ultimate] (1). R. van Elst <remy@example2.nl>

Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu


sec  secp256k1/A2EB77DDEA029199
     created: 2018-06-01  expires: 2019-06-01  usage: SC  
     trust: ultimate      validity: ultimate
ssb  secp256k1/770B8C8E75662ABE
     created: 2018-06-01  expires: 2019-06-01  usage: E   
[ultimate] (1). R. van Elst <remy@example2.nl>


sec  secp256k1/A2EB77DDEA029199
     created: 2018-06-01  expires: 2019-06-01  usage: SC  
     trust: ultimate      validity: ultimate
ssb  secp256k1/770B8C8E75662ABE
     created: 2018-06-01  expires: 2019-06-01  usage: E   
[ultimate] (1). R. van Elst <remy@example2.nl>

The keys are now marked as trusted in the local trust database. We can continue on to sign them.

Signing the keys noninteractively in batch mode

Signing the keys tells other people that we verified the identity of the key owners and trusting their keys, confirming that with a signature of our own key. Because we all were in a room together doing this I did verify their identity, thus vouching for their public keys.

This command signs all the keys found in the keyring:

$ for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u); do echo -e "y\ny\n" |  gpg --command-fd 0 --expert --edit-key $fpr sign; done

Depending on how your key agent is set up it will prompt you on the command line for the passphrase, or a GUI dialog window.

Output looks like this:

gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa2048/2B6755BD1B7F88DC
     created: 2014-06-01  expires: never       usage: SCEA
     trust: ultimate      validity: ultimate
sub  rsa2048/CE2786EB97AC7685
     created: 2014-06-01  expires: 2019-05-31  usage: E   
[ultimate] (1). Remy van Elst <remy@example3.net>

User ID "Remy van Elst <remy@example3.net>" is not self-signed.

pub  rsa2048/2B6755BD1B7F88DC
     created: 2014-06-01  expires: never       usage: SCEA
     trust: ultimate      validity: ultimate
 Primary key fingerprint: 4DDE 73DB 5030 B539 2681  3A50 2B67 55BD 1B7F 88DC

     Remy van Elst <remy@example3.net>

Are you sure that you want to sign this key with your
key "R. van Elst <remy@example2.nl>" (A2EB77DDEA029199)



pub  rsa2048/2B6755BD1B7F88DC
     created: 2014-06-01  expires: never       usage: SCEA
     trust: ultimate      validity: ultimate
sub  rsa2048/CE2786EB97AC7685
     created: 2014-06-01  expires: 2019-05-31  usage: E   
[ultimate] (1). Remy van Elst <remy@example3.net>

If you want to use a specific key to sign with, for example when you hae more then 1 private key, add the --local-user parameter:

$ echo "y\ny\n" | gpg --command-fd 0 --expert --local-user BA3185A7E50F713280F4559AA2EB77DDEA029199 --edit-key 4DDE73DB5030B53926813A502B6755BD1B7F88DC  sign

Publishing the keys to a keyserver noninteractive in batch mode

Now that all keys are trusted and signed we can publish the result to a keyserver so that we can tell the entire world about these new trust relations. You probably are able to guess the command, it uses the same loop as before:

$ for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u); do gpg --send-keys --keyserver pool.sks-keyservers.net $fpr; done

Output looks like below:

gpg: sending key 6F8ACF16C411318B to hkp://pool.sks-keyservers.net
gpg: sending key A2EB77DDEA029199 to hkp://pool.sks-keyservers.net
gpg: sending key 2B6755BD1B7F88DC to hkp://pool.sks-keyservers.net
gpg: sending key 2B6755BD1B7F88DC to hkp://pool.sks-keyservers.net
gpg: sending key 39110B67C1165E3F to hkp://pool.sks-keyservers.net
gpg: sending key 39110B67C1165E3F to hkp://pool.sks-keyservers.net
gpg: sending key A2EB77DDEA029199 to hkp://pool.sks-keyservers.net
gpg: sending key 3A624724EAFF8D71 to hkp://pool.sks-keyservers.net
gpg: sending key 3A624724EAFF8D71 to hkp://pool.sks-keyservers.net
gpg: sending key 6F8ACF16C411318B to hkp://pool.sks-keyservers.net

That's all there is to it. Some shell commands chained together to save me a lot of time and effort. Just the way I like it.

License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source.

This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it.

All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software:

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Just to be clear, the information on this website is for meant for educational purposes and you use it at your own risk. I do not take responsibility if you screw something up. Use common sense, do not 'rm -rf /' as root for example. If you have any questions, do not hesitate to contact me.

See https://raymii.org/s/static/About.html for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment