Freepbx Voicemail Transcription Script: Google Speech API

sendmail-gcloud

#!/bin/sh

# sendmail-gcloud
#
# Installation instructions
# Copy the content of this file to /usr/sbin/sendmail-gcloud
#
# Google Account
# ---------------
# Create a Google Cloud account if you don't have one yet. Free trial is available at https://console.cloud.google.com/freetrial
# Within console.cloud.google.com search for Cloud Speech-to-Text API and enable it
# Some users report you need to have configured a service account: See creating a service account in Google Cloud. https://cloud.google.com/iam/docs/keys-create-delete#creating
#
# From the Linux command line on the FreePBX machine
# -------------------------------------------
# Follow steps 1 and 3 of the instructions on Google Cloud https://cloud.google.com/sdk/docs/downloads-yum
#   Step 1 Note: since FreePBX is Centos 7, follow the instructions to replace el8 with el7 in the base url:
#   sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM [google-cloud-cli] name=Google Cloud CLI baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-e17-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOM
#
#   Step 3 Note: use yum instead of dnf:
#   yum install google-cloud-cli
#
# Run the following commands on FreePBX;
#    cd /usr/sbin/
#    chown asterisk:asterisk sendmail-gcloud
#    chmod 744 sendmail-gcloud
#    chmod 744 /usr/bin/dos2unix
#
# Verify that you have the following (by simply running the command) and if not use yum install;
#    jq
#    sox
#    flac
#    dos2unix -V
# Ensure dos2unix is executable by the asterisk user (chmod 777 /usr/bin/dos2unix)
#
# Connect FreePBX to Google Cloud
#     su asterisk
#     gcloud auth login
# CLI will provide you a url. Copy that and paste it into your browser. Google will give you a verification code to copy.
# Paste it into the cli waiting for a verification code.
#
# Some users report that you need to run the following at this point:
#     gcloud config set project "Your Project ID"
#
# Open FreePBX web interface
#     Go to Settings > Voicemail Admin > Settings > Email Config
#     Change Mail Command to: /usr/sbin/sendmail-gcloud
#     Submit and apply changes
#
# Original source created by N. Bernaerts: https://github.com/NicolasBernaerts/debian-scripts/tree/master/asterisk
# modified per: https://jrklein.com/2015/08/17/asterisk-voicemail-transcription-via-ibm-bluemix-speech-to-text-api/
# modified per: https://gist.github.com/lgaetz/2cd9c54fb1714e0d509f5f8215b3f5e6
# current version: https://gist.github.com/tony722/7c6d86be2e74fa10a1f344a4c2b093ea
#
# Notes: This is a script modified from the original to work with FreePBX so that email notifications sent from
#        Asterisk voicemail contain a speech to text transcription provided by Google Cloud Speech API
#
# License: There are no explicit license terms on the original script or on the blog post with modifications
#          I'm assumig GNU/GPL2+ unless notified otherwise by copyright holder(s)
#
# Version History:
# 2023-12-11  Add gcloud cli parameters by grintor to enhance gcloud ml telephony transcription
# 2023-09-01  Update instructions for installing google-cloud-cli
# 2023-08-24  Add fix by EagleTalonSystems: gcloud config set project "Your Project ID"
# 2021-05-06  Add fix by dcat127: trim flac file to 59 seconds
# 2020-08-27  Add fix by chrisduncansn
#             Minor edit in instruction wording
# 2020-05-27  Add instructions from sr10952
#             Add export fix by levishores 
# 2019-02-27  Initial commit by tony722
# set PATH
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# save the current directory
pushd .
 
# create a temporary directory and cd to it
TMPDIR=$(mktemp -d)
cd $TMPDIR
 
# dump the stream to a temporary file
cat >> stream.org
 
# get the boundary
BOUNDARY=$(grep "boundary=" stream.org | cut -d'"' -f 2)
 
# if mail has no boundaries, assume no attachment
if [ "$BOUNDARY" = "" ]
	
then
	# send the original stream
	mv stream.org stream.new
else
	# cut the original stream into parts
	# stream.part  - header before the boundary
	# stream.part1 - header after the bounday
	# stream.part2 - body of the message
	# stream.part3 - attachment in base64 (WAV file)
	# stream.part4 - footer of the message
	awk '/'$BOUNDARY'/{i++}{print > "stream.part"i}' stream.org
 
	# cut the attachment into parts
	# stream.part3.head - header of attachment
	# stream.part3.wav.base64 - wav file of attachment (encoded base64)
	sed '7,$d' stream.part3 > stream.part3.wav.head
	sed '1,6d' stream.part3 > stream.part3.wav.base64

	# convert the base64 file to a wav file
	dos2unix -o stream.part3.wav.base64
	base64 -di stream.part3.wav.base64 > stream.part3.wav

	# convert the wav file to FLAC
	sox -G stream.part3.wav --channels=1 --bits=16 --rate=8000 stream.part3.flac trim 0 59

	# convert to MP3
	sox stream.part3.wav stream.part3-pcm.wav
	lame -m m -b 24 stream.part3-pcm.wav stream.part3.mp3 
	base64 stream.part3.mp3 > stream.part3.mp3.base64

	# create mp3 mail part
	sed 's/x-[wW][aA][vV]/mpeg/g' stream.part3.wav.head | sed 's/.[wW][aA][vV]/.mp3/g' > stream.part3.new
	dos2unix -o stream.part3.new 
	unix2dos -o stream.part3.mp3.base64
	cat stream.part3.mp3.base64 >> stream.part3.new

	# save voicemail in tmp folder in case of trouble
	# TMPMP3=$(mktemp -u /tmp/msg_XXXXXXXX.mp3)
	# cp "stream.part3.mp3" "$TMPMP3"

	export CLOUDSDK_CONFIG=/home/asterisk/.config/gcloud
	RESULT=`gcloud ml speech recognize stream.part3.flac --language-code='en-US' --model=phone_call --filter-profanity --enable-automatic-punctuation`

	FILTERED=`echo "$RESULT" | jq -r '.results[].alternatives[].transcript'`
	 	   
	# generate first part of mail body, converting it to LF only
	mv stream.part stream.new
	cat stream.part1 >> stream.new
	sed '$d' < stream.part2 >> stream.new

	# beginning of transcription section
	echo "" >> stream.new
	echo "--- Google transcription result ---" >> stream.new

	# append result of transcription
	if [ -z "$FILTERED" ]
	then
	  echo "(Google was unable to recognize any speech in audio data.)" >> stream.new
	else
	  echo "$FILTERED" >> stream.new
	fi

	# end of message body
	tail -1 stream.part2 >> stream.new

	# add converted attachment  
	cat stream.part3.new >> stream.new

	# append end of mail body, converting it to LF only
	echo "" >> stream.tmp
	echo "" >> stream.tmp
	cat stream.part4 >> stream.tmp
	dos2unix -o stream.tmp
	cat stream.tmp >> stream.new
 
fi
 
# send the mail thru sendmail
cat stream.new | sendmail -t
 
# go back to original directory
popd
 
# remove all temporary files and temporary directory
rm -Rf $TMPDIR

@tony722 Thanks for adding that.

One other thing that might confuse people, google might of changed either webpage but for:

"# Follow steps 1 and 2 of the instructions on Google Cloud https://cloud.google.com/sdk/docs/downloads-yum"

It should be 1 and 3, since the FreePBX distro is Centos version 7.

Step 1
'
sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM
[google-cloud-cli]
name=Google Cloud CLI
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el8-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM
'

Step 3
yum install google-cloud-cli

Everything else is perfect

I see that Google transcribe can add punctuation. Can the script the modified to pass that variable? https://cloud.google.com/speech-to-text/docs/automatic-punctuation

I just installed this and it works a treat. A genius idea with very clear instructions. Thank you.

EDIT: changing the code line 130 as below improved the transcript quality:

FROM:
gcloud ml speech recognize stream.part3.flac --language-code='en-US'
TO:
gcloud alpha ml speech recognize-long-running stream.part3.flac --language-code='en-US' --enable-automatic-punctuation --interaction-type='voicemail' --model='phone_call_enhanced'

For better dictation, you should change the command from
gcloud ml speech recognize stream.part3.flac --language-code='en-US'
to
gcloud ml speech recognize stream.part3.flac --language-code='en-US' --enable-automatic-punctuation --model=phone_call_enhanced

I was toying around with this on FreePBX 17 which is now based on Debian 12

I've taken into account every previous issue reported however I still only see the following from the emails which are processed:
--- Google transcription result ---
(Google was unable to recognize any speech in audio data.)

any advice on how to further troubleshoot ?

This was terrific - thanks! A couple of thoughts... Some of these might only apply to my situation - I installed Asterisk from the repo, not with FreePBX:

1. Add 'lame' to the prerequisites
2. I reworked the transcription, which didn't work for me as originally coded:

export CLOUDSDK_CONFIG=/home/myid/.config/gcloud
RESULT=`gcloud ml speech recognize stream.part3.flac --language-code='en-US' --model=phone_call --filter-profanity --enable-automatic-punctuation 2> transcribe_error.txt`
echo $RESULT > transcription.txt
FILTERED=`jq -r '.results[].alternatives[].transcript' < transcription.txt`

Piping stderr to a file identified gcloud authentication problems, and saving the transcription text really boosted my morale!

My asterisk user is marked "no login" ("This account is currently not available"), maybe because of the way Asterisk was installed. Anyway, I couldn't figure out an easy way to authenticate the asterisk user with gcloud, so I had used my own login account's authentication:
export CLOUDSDK_CONFIG=/home/myuserid/.config/gcloud
and did chown -R asterisk:asterisk. In retrospect, it might have been better to temporarily set asterisk to the bash shell, do the gcloud auth thing, then set it back to nologin - but it now works so I'm leaving it alone.