Freepbx Voicemail Transcription Script: Google Speech API

sendmail-gcloud

#!/bin/sh

# sendmail-gcloud
#
# Installation instructions
# Copy the content of this file to /usr/sbin/sendmail-gcloud
#
# Google Account
# ---------------
# Create an account, if you don't have yet, with Google Cloud. Free trials can be made at https://console.cloud.google.com/freetrial
# Within console.cloud.google.com search for Cloud Speech-to-Text API and enable it
#
# From the Linux command line on the FreePBX machine
# -------------------------------------------
# Follow steps 1 and 2 of the instructions on Google Cloud https://cloud.google.com/sdk/docs/downloads-yum
# Run the following commands on FreePBX;
#    cd /usr/sbin/
#    chown asterisk:asterisk sendmail-gcloud
#    chmod 744 sendmail-gcloud
#    chmod 777 /usr/bin/dos2unix
#
# Verify that you have the following (by simply running the command) and if not use yum install;
#    jq
#    sox
#    flac
#    dos2unix -V
# Ensure dos2unix is executable by the asterisk user (chmod 777 /usr/bin/dos2unix)
#
# Connect FreePBX to Google Cloud
#     su asterisk
#     gcloud auth login
# CLI will provide you a url. Copy that and paste it into your browser. Google will give you a verification code to copy. Paste it into the cli waiting for a verification code.
#
# Open FreePBX web interface
#     Go to Settings > Voicemail Admin > Settings > Email Config
#     Change Mail Command to: /usr/sbin/sendmail-gcloud
#     Submit and apply changes
#
# Original source created by N. Bernaerts: https://github.com/NicolasBernaerts/debian-scripts/tree/master/asterisk
# modified per: https://jrklein.com/2015/08/17/asterisk-voicemail-transcription-via-ibm-bluemix-speech-to-text-api/
# modified per: https://gist.github.com/lgaetz/2cd9c54fb1714e0d509f5f8215b3f5e6
# current version: https://gist.github.com/tony722/7c6d86be2e74fa10a1f344a4c2b093ea
#
# Notes: This is a script modified from the original to work with a FreePBX Distro PBX so that email notifications sent from
#        Asterisk voicemail contain a a speech to text transcription provided by Google Cloud Speech API
#
# License: There are no explicit license terms on the original script or on the blog post with modifications
#          I'm assumig GNU/GPL2+ unless notified otherwise by copyright holder(s)
#
# Version History:
# 2020-05-27  Add instructions from sr10952
#             Add export fix by levishores 
# 2019-02-27  Initial commit by tony722

# set PATH
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# save the current directory
pushd .
 
# create a temporary directory and cd to it
TMPDIR=$(mktemp -d)
cd $TMPDIR
 
# dump the stream to a temporary file
cat >> stream.org
 
# get the boundary
BOUNDARY=$(grep "boundary=" stream.org | cut -d'"' -f 2)
 
# if mail has no boundaries, assume no attachment
if [ "$BOUNDARY" = "" ]
	
then
	# send the original stream
	mv stream.org stream.new
else
	# cut the original stream into parts
	# stream.part  - header before the boundary
	# stream.part1 - header after the bounday
	# stream.part2 - body of the message
	# stream.part3 - attachment in base64 (WAV file)
	# stream.part4 - footer of the message
	awk '/'$BOUNDARY'/{i++}{print > "stream.part"i}' stream.org
 
	# cut the attachment into parts
	# stream.part3.head - header of attachment
	# stream.part3.wav.base64 - wav file of attachment (encoded base64)
	sed '7,$d' stream.part3 > stream.part3.wav.head
	sed '1,6d' stream.part3 > stream.part3.wav.base64

	# convert the base64 file to a wav file
	dos2unix -o stream.part3.wav.base64
	base64 -di stream.part3.wav.base64 > stream.part3.wav

	# convert the wav file to FLAC
	sox -G stream.part3.wav --channels=1 --bits=16 --rate=8000 stream.part3.flac

	# convert to MP3
	sox stream.part3.wav stream.part3-pcm.wav
	lame -m m -b 24 stream.part3-pcm.wav stream.part3.mp3 
	base64 stream.part3.mp3 > stream.part3.mp3.base64

	# create mp3 mail part
	sed 's/x-[wW][aA][vV]/mpeg/g' stream.part3.wav.head | sed 's/.[wW][aA][vV]/.mp3/g' > stream.part3.new
	dos2unix -o stream.part3.new 
	unix2dos -o stream.part3.mp3.base64
	cat stream.part3.mp3.base64 >> stream.part3.new

	# save voicemail in tmp folder in case of trouble
	# TMPMP3=$(mktemp -u /tmp/msg_XXXXXXXX.mp3)
	# cp "stream.part3.mp3" "$TMPMP3"

	export CLOUDSDK_CONFIG=/home/asterisk/.config/gcloud
	RESULT=`gcloud ml speech recognize stream.part3.flac --language-code='en-US'`

	FILTERED=`echo "$RESULT" | jq -r '.results[].alternatives[].transcript'`
	 	   
	# generate first part of mail body, converting it to LF only
	mv stream.part stream.new
	cat stream.part1 >> stream.new
	sed '$d' < stream.part2 >> stream.new

	# beginning of transcription section
	echo "" >> stream.new
	echo "--- Google transcription result ---" >> stream.new

	# append result of transcription
	if [ -z "$FILTERED" 	]
	then
	  echo "(Google was unable to recognize any speech in audio data.)" >> stream.new
	else
	  echo "$FILTERED" >> stream.new
	fi

	# end of message body
	tail -1 stream.part2 >> stream.new

	# add converted attachment  
	cat stream.part3.new >> stream.new

	# append end of mail body, converting it to LF only
	echo "" >> stream.tmp
	echo "" >> stream.tmp
	cat stream.part4 >> stream.tmp
	dos2unix -o stream.tmp
	cat stream.tmp >> stream.new
 
fi
 
# send the mail thru sendmail
cat stream.new | sendmail -t
 
# go back to original directory
popd
 
# remove all temporary files and temporary directory
rm -Rf $TMPDIR

I have been attempting to make this work for the last 4 hours and getting nowhere. I know I have google cloud set up correctly because I can change the TMPDIR to an actual location and have the system write the files there then copy the "gcloud ml speech recognize stream.part3.flac --language-code='en-US'" command and run it in that dir as the asterisk user and get the JSON however it is not running as part of the script.

This is code that I'm using on a production FreePBX server (just diffed it to be sure) . Keep trying, checking logs, messages, permissions, and whatever else. Hopefully you can get it. 😄 Good luck!

I forgot to post back I decided to try a stupid this morning and "reboot -h now" I left a vm after the system restarted and it transcribed properly.

Thanks a lot for your great work.
Could you please make this work with VitalPBX?
This is working good with IBM Watson (example):
http://incrediblepbx.com/sendmailibm-vitalpbx.tar.gz
Doc here: https://nerdvittles.com/?p=25697

But i would love this do work with Google Speech to Text.

Tried copy/paste your script but i only get:
--- Google transcription result ---
(Google was unable to recognize any speech in audio data.)

That's a great idea @pete1019. However I'm not running VitalPBX so I don't have a way to do this. I maintain a single FreePBX system, and just put this out there in case it helps anyone else. :-) I will say that I get this message from Google occasionally. However if you get it every time there's likely a problem.

Feel free to fork this and make any changes needed. Good luck!

I think it is just a owner / rights problem.
Everything works but nothing happens here when your script is used here:
RESULT=gcloud ml speech recognize stream.part3.flac --language-code='en-US'

If i run this command as root everyhing is fine and i get results.
How should i set rights and ownership in CentOS since you said:
"gcloud utility must be authenticated. Before doing this, 'su asterisk' so authentication happens in the correct user account"
I did this as root cause i did not know any better.
If i do su asterisk it says: This account is currently not available. Maybe since user asterisk has nologin? Solution?
THANK YOU!

I've never looked at VItalPBX at all, so I don't know what user accounts it runs under. I'd suggest taking this script to the guys on the VitalPBX forums and see if they can help--I'm really unable to help at all! 😄 Good luck!

"gcloud utility must be authenticated. Before doing this, 'su asterisk' so authentication happens in the correct user account"
This does the trick:
su -s /bin/bash asterisk

I've been using this script for some time now (with FreePBX 14), and was previously using the IBM Bluemix version with no issues. However lately I've noticed that every several weeks I'll start getting the "Google was unable to recognize any speech in audio data." result. But if I pipe a raw message through the script as the asterisk user it works perfectly fine. It seems the issue only occurs when the script is called by FreePBX. Only a reboot seems to solve it. Dug through the mail/asterisk/gcloud logs and I'm not seeing anything useful. Any ideas?

Might be good to ask on the FreePBX forums. I've seen that kind of issue randomly, but it's always the odd message here or there and never required a reboot. I'm not even pretending to be an expert on all this. But this script worked for me so I posted it here. 😄 Please post back a fix here if you find one! :-)

FYI, we found that the cause of the periodic failure of this script was caused by the Script attempting to get the gcloud config from /root/.config/gcloud. Seems like the parent process's HOME environment variable is used by gcloud. I added
export CLOUDSDK_CONFIG=/home/asterisk/.config/gcloud
above
RESULT=gcloud ml speech recognize stream.part3.flac --language-code='en-US'``
which corrected the issue for us without a reboot.

Time will tell if this keeps up.

@levishores Makes total sense, good find! I've implemented your fix and it worked.

Also, for posterity, you can debug the gcloud command by adding this commented line:

//Insert this line to use for debugging - will dump console errors to 'error' in the tmp dir
gcloud ml speech recognize stream.part3.flac --language-code='en-US' 1>result 2>error
//put it near this line in the script (don't edit this line):
RESULT=`gcloud ml speech recognize stream.part3.flac --language-code='en-US'`

//comment this line at the bottom to keep the TMP directory for analysis after the script runs
rm -Rf $TMPDIR

Hope this makes sense...
The concept is

1. Comment the line at the bottom which deletes the temp directory - this lets you go into /tmp/tmp.[varies] to analyze the results
2. adding a second gcloud command with '1>result 2>error' at the end results in console errors going to 'error' file. (This is the critical part of tracking down why the script isn't executing properly when called by Asterisk. Errors don't feed into the variable.
3. Once changed, leave a VM, then navigate to /tmp/tmp.[uid] and investigate the error file

Thanks a lot for your code. I am in the process of setting it up. While setting it up I have actually tried to create a helpful user guide for, less technical people to follow along. After much troubleshooting, my FreePBX is talking to google... However, the email received is saying "Google was unable to recognize any speech in audio data."

Looking through the "result" file in the log, I do see results being sent back from Google. below are two different "results" I received, from two separate voicemails. What is my next step on here?

{ "results": [ { "alternatives": [ { "confidence": 0.818743, "transcript": "hi, this is just a test message to see if" } ] }, { "alternatives": [ { "confidence": 0.6924237, "transcript": " Google being able to transcribe this message." } ] } ] }

Thinking the issue is with two levels of confidence within the result, I left another message, which was also not passed down in the email body.

{ "results": [ { "alternatives": [ { "confidence": 0.73869956, "transcript": "Hi, this message is for test how Google will be able to transcribe this voicemail and send it back to me." } ] } ] }
edit;
ok, figured it out

It looks like a change Google made may have broken this script. Do you have a fix that I could incorporate here?

Tony, I would love to be able to help you with this.
The issues I had is not something that would "break". Are you not getting back from Google anything? The issue I had in the previous comment was due to some troubleshooting I took. Perhaps this is the case by you as well. Basically, on line 89 of your current code above, I had changed it based on @levishores comment. I changed that line to read
RESULT= gcloud ml speech recognize stream.part3.flac --language-code='en-US' 1>result 2>error. this broke everything. I changed it back to the original,
gcloud ml speech recognize stream.part3.flac --language-code='en-US' and now it seems to work perfectly.

Steps needed to get FreePBX to work with Google Cloud Speech to Text for voicemail transcripts

Google Account

1. Create an account, if you don't have yet, with Google Cloud. Free trials can be made at https://console.cloud.google.com/freetrial
2. Within console.cloud.google.com search for Cloud Speech-to-Text API and enable it

Within FreePBX

1. Follow steps 1 and 2 of the instructions on Google Cloud https://cloud.google.com/sdk/docs/downloads-yum
2. Copy the content of the code on this page to /usr/sbin/sendmail-gcloud
3. Run the following commands on FreePBX;
   cd /usr/sbin/
chown asterisk:asterisk sendmail-gcloud
chmod 744 sendmail-gcloud
chmod 777 /usr/bin/dos2unix
4. Verify that you have the following (by simply running the command) and if not use yum install;
   jq
   sox
   flac

Connect FreePBX to Google Cloud

1. Within FreePBX run the following;
   su asterisk
gcloud auth login
   CLI will provide you a url. Copy that and paste it into your browser. Google will give you a verification code to copy. Paste it into the cli waiting for a verification code.

Have FreePBX inject this code
Within FreePBX go to Settings > Voicemail Admin > Settings > Email Config, and add “/usr/sbin/sendmail-gcloud” (without the quotes) into the Mail Command. Submit and apply changes

Hopefully this should do get it to work

I have updated my comment. Github screwed with the ticks in the code, so yeah, if you just blindly copied and pasted
RESULT= gcloud ml speech recognize stream.part3.flac --language-code='en-US' 1>result 2>error
you were gonna break the script. (I'll also add, this isn't even the line I suggested you add - you're setting the RESULT variable equal to the line that logs the output to two different files. Not even sure what bash is going to do there, but I'll bet it's not gonna work.)

Did you read that full comment? The concept is using it for debugging because the gcloud command doesn't execute properly when it's executed by Asterisk as part of the script. You can't see the error, but if you add an additional gcloud command and log the error text to a file, then keep the script from deleting the TMP file, you can look at the error. Root cause was that it was trying to access the gcloud config inside the root home folder, not asterisk's home.

The fix was simply to add
export CLOUDSDK_CONFIG=/home/asterisk/.config/gcloud
anywhere above the gcloud command.

I have actually been able to debug it by only stopping deletion of the temp files, so I was able to see somewhat at which step it stops, although it was much more difficult to troubleshoot... I wrote this guide above that actually worked for me in production. (latest verison of FreePBX and gcloud).

@sr10952. I've incorporated your instructions into the original gist as comments so they can be seen too. Thanks!

@levishores, I've added the export line to the gist too. Thanks for your contrib. :-)