Generate an x509 certificate and a TLSA record with openssl

x509-dane.md

Creating a self-signed certificate for example.com (if you already have a certificate you can skip this step):

openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
  -keyout cert.key -out cert.crt -extensions ext  -config \
  <(echo "[req]"; 
    echo distinguished_name=req; 
    echo "[ext]";
    echo "keyUsage=critical,digitalSignature,keyEncipherment";
    echo "extendedKeyUsage=serverAuth";
    echo "basicConstraints=critical,CA:FALSE";
    echo "subjectAltName=DNS:example.com,DNS:*.example.com";
    ) -subj "/CN=*.example.com"

Generate the TLSA record rdata (you can also use this tool):

echo -n "3 1 1 " && openssl x509 -in cert.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | xxd  -p -u -c 32

Add the TLSA record to your zone file

Using a remote host

echo -n "3 1 1 " && openssl s_client -connect mx1.mails.example.org:465 2>/dev/null | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | xxd  -p -u -c 32

And to test it:

# Did not work for me
openssl s_client -brief -starttls smtp -dane_tlsa_domain mx1.mails.example.org -dane_tlsa_rrdata "3 1 1 666792F7857D333D59655A19CA59CD88B7C535CD699DCEAA290B943DEB8E23FD" -connect mx1.mails.example.org:25 <<< "Q"
# Works for me
openssl s_client -brief -dane_tlsa_domain mx1.mails.example.org -dane_tlsa_rrdata "3 1 1 666792F7857D333D59655A19CA59CD88B7C535CD699DCEAA290B943DEB8E23FD" -connect mx1.mails.example.org:465 <<< "Q"

Ref: https://github.com/internetstandards/toolbox-wiki/blob/main/DANE-for-SMTP-how-to.md
Ref: https://www.mailhardener.com/kb/how-to-create-a-dane-tlsa-record-with-openssl

The best is to test on: https://www.checktls.com/TestReceiver

To get the TLSA record of a remote host (updated):

echo -n "3 1 1 " && openssl s_client -connect 157.245.63.29:443 -servername letsdane <<< "Q" 2>/dev/null | openssl x509 -pubkey -noout -in /dev/stdin | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | xxd -p -u -c 32

• add -in /dev/stdin to remove openssl's new warning that reads Warning: Reading certificate from stdin since no -in or -new option is given
• add <<< "Q" so the connection is quit automaticlly without a ctrl+C
• optionally separate out IP with -servername

To use for locally available certificate (cert.pem in current directory for example) you can just use this simplified command:

echo -n "3 1 1 " && openssl x509 -pubkey -noout -in cert.pem | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | xxd -p -u -c 32

Thanks @rithvikvibhu for providing original command, it helped me a lot!