Skip to content

Instantly share code, notes, and snippets.

@EverythingSmartHome
Last active April 23, 2024 03:23
Show Gist options
  • Star 40 You must be signed in to star a gist
  • Fork 5 You must be signed in to fork a gist
  • Save EverythingSmartHome/055fbdde31a607ef9d695d5cac780e94 to your computer and use it in GitHub Desktop.
Save EverythingSmartHome/055fbdde31a607ef9d695d5cac780e94 to your computer and use it in GitHub Desktop.
ESP32 & ESPHome Voice Assistant
esphome:
name: esp32-mic-speaker
friendly_name: esp32-mic-speaker
on_boot:
- priority: -100
then:
- wait_until: api.connected
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
esp32:
board: esp32dev
framework:
type: esp-idf
version: recommended
# Enable logging
logger:
# Enable Home Assistant API
api:
ota:
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "Esp32-Mic-Speaker"
password: "9vYvAFzzPjuc"
i2s_audio:
i2s_lrclk_pin: GPIO27
i2s_bclk_pin: GPIO26
microphone:
- platform: i2s_audio
id: mic
adc_type: external
i2s_din_pin: GPIO13
pdm: false
speaker:
- platform: i2s_audio
id: big_speaker
dac_type: external
i2s_dout_pin: GPIO25
mode: mono
voice_assistant:
microphone: mic
use_wake_word: false
noise_suppression_level: 2
auto_gain: 31dBFS
volume_multiplier: 2.0
speaker: big_speaker
id: assist
switch:
- platform: template
name: Use wake word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
entity_category: config
on_turn_on:
- lambda: id(assist).set_use_wake_word(true);
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
on_turn_off:
- voice_assistant.stop
- lambda: id(assist).set_use_wake_word(false);
@rich33584
Copy link

@rich33584 That config seems to be working for me. Im still hitting some issues with TTS getting cut short or the wake work getting picked up by STT. But all in all its actually working better.

I seem to have spells of hours that it works perfectly, and then hours where the TTS is broken and crappy.
Not sure if its a wifi issue or if the ESP32 is just barely adequate to run these. I ordered som "Better" esp32 modules. Ill report back on them.

@Sn00kiT
Copy link

Sn00kiT commented Feb 27, 2024

Have you seen this?
https://beta.esphome.io/components/micro_wake_word.html

Has anyone luck with an ESP32-S3? The device seems to be more capable to manage the workload

In case someone needs a working config without crackling speakers in correct formating.

esphome:
  name: ha-mic-speaker01
  friendly_name: ha-mic-speaker01

esp32:
  board: esp32dev
  framework:
    type: esp-idf

# Enable logging
logger:

web_server:

# Enable Home Assistant API
api:
  encryption:
    key: "paste_your_key_here"

ota:
  password: "paste_your_key_here"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Ha-Mic-Speaker01"
    password: "paste_your_key_here"

captive_portal:

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO26   #WS / LRC
    i2s_bclk_pin: GPIO25    #SCK /BCLK

microphone:
  - platform: i2s_audio
    adc_type: external
    pdm: false
    id: mic_i2s
    channel: right
    bits_per_sample: 32bit
    i2s_audio_id: i2s_in
    i2s_din_pin: GPIO14    #SD

speaker:
  - platform: i2s_audio
    id: my_speaker
    dac_type: external
    i2s_dout_pin: GPIO12   #DIN 
    mode: mono
    i2s_audio_id: i2s_in


voice_assistant:
  microphone: mic_i2s
  id: va
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 4.0
  use_wake_word: false
  speaker: my_speaker
  
  on_error: 
   - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - switch.turn_off: use_wake_word
          - switch.turn_on: use_wake_word      

  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:

  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
  

binary_sensor:
  - platform: status
    name: API Connection
    id: api_connection
    filters:
      - delayed_on: 1s
    on_press:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.start_continuous:
    on_release:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.stop:


switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);

@jake8796
Copy link

PS D:\Projects\LocalVoiceAssistant> esphome run voiceAssistantESP32S3.yaml
INFO ESPHome 2024.2.2
INFO Reading configuration voiceAssistantESP32S3.yaml...
INFO Updating https://github.com/esphome/esphome.git@pull/5230/head
Failed config

media_player.i2s_audio: [source voiceAssistantESP32S3.yaml:74]

This feature is only available with frameworks ['arduino'].
platform: i2s_audio
name: esp_speaker
id: media_player_speaker
dac_type: external
i2s_audio_id: i2s_out
i2s_dout_pin: GPIO16
mode: mono

Any ideas why I might be getting this error?

@LumCrafter
Copy link

INFO Resolving IP address of esp32-mic-speaker.local
ERROR Error resolving IP address of esp32-mic-speaker.local. Is it connected to WiFi?
ERROR (If this error persists, please set a static IP address: https://esphome.io/components/wifi.html#manual-ips)
ERROR Error resolving IP address: Error resolving address with mDNS: Did not respond. Maybe the device is offline., [Errno -5] No address associated with hostname

I keep getting this error what do i do?

@Xornop
Copy link

Xornop commented Mar 27, 2024

my setup has a screeching sound instead of a crackle when playing back a response. using the setup posted above but using different gpio pins.

Any ideas?

@Djelle
Copy link

Djelle commented Mar 29, 2024

I have discovered, that many of the problem mentioned above is due to interference noise. I had a lot of crackling noise with 10 cm wires on my test setup. So I had to go study on the net.

I2S is ment to be used between components on a PCB. It is without error protocol, so any errors will result in some kind of static noise. The solution is to keep the I2S connection as short as possible. There is no room for the pins on the ESP32 in my box. So I cut those off. Though I left the plastic part of the pin-row. I clammed the MAX98357A board on the backside of the ESP32 PCB, so I only needed 5 mm of wire. I placed the INMP441 on the side-edge of the ESP32 so they form a T (because that works with the box i use). And now it works without any noise.

I don't know how long the wires can be without shielding. But if you need to have even longer wires, they must be individually shielded (only the I2S wires). Shields should be connected to ground on the ESP32 (so only connected in one end). I have no idea how long they then can be. But not especially long. Maybe 20-30 cm?

/Djelle

@Battman2013
Copy link

Battman2013 commented Mar 29, 2024

PS D:\Projects\LocalVoiceAssistant> esphome run voiceAssistantESP32S3.yaml INFO ESPHome 2024.2.2 INFO Reading configuration voiceAssistantESP32S3.yaml... INFO Updating https://github.com/esphome/esphome.git@pull/5230/head Failed config

media_player.i2s_audio: [source voiceAssistantESP32S3.yaml:74]

This feature is only available with frameworks ['arduino']. platform: i2s_audio name: esp_speaker id: media_player_speaker dac_type: external i2s_audio_id: i2s_out i2s_dout_pin: GPIO16 mode: mono

Any ideas why I might be getting this error?

As the error message tells, the media_player is available in the arduino framework only.

esp32:
board: esp32dev # board type
framework:
type: arduino

@edwardtich1
Copy link

edwardtich1 commented Apr 9, 2024

The sound is interrupted when playing through the speaker

here is my code

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO26
    i2s_bclk_pin: GPIO25

media_player:
  - platform: i2s_audio
    name: "Informer"
    id: media_player_speaker
    i2s_audio_id: i2s_in
    dac_type: external
    i2s_dout_pin: GPIO27
    mode: mono
    on_pause:
      - media_player.stop

here is my log

13:32:01	[W]	[component:232]	
Component i2s_audio.media_player took a long time for an operation (167 ms).
13:32:01	[W]	[component:233]	
Components should block for at most 30 ms.
13:32:01	[W]	[component:233]	
Components should block for at most 30 ms.

please help!

@imonlinux
Copy link

imonlinux commented Apr 17, 2024

I was doing some serious google'ing and ran across this Github repo: https://github.com/gnumpi/esphome_audio

Using a lot of what was put in this thread and the work that gnumpi has done on media-player for esp-idf, I have put together this config running on an esp-wroom-32. Still testing (and I think that I'm going to need a S3 board to make this reliable) but I did get it to work last night with a mic and speaker (4Ohm, 3W) with great sound.

Next I am going to add a 16bit led ring and see if I can get even better success on an S3 board.

substitutions:
  device_name: test-media-assistant
  friendly_name: Test Assistant
  device_description: "ESP WROOM 32"
  api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

external_components:
  - source:
      type: git
      url: https://github.com/gnumpi/esphome_audio
      ref: main
    components: [ adf_pipeline, i2s_audio ]

esphome:
  name: ${device_name}
  comment: ${device_description}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf

logger:

api:
  encryption:
    key: ${api_key}

ota:
  password: !secret ota_password

wifi:
  enable_rrm: true
  ssid: !secret wifi_ssid
  password: !secret wifi_password

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO21 # INMP441 WS 
    i2s_bclk_pin: GPIO22  # INMP441 SCK 
  - id: i2s_out
    i2s_lrclk_pin: GPIO25 # PCM5102 LCK 
    i2s_bclk_pin: GPIO26  # PCM5102 BCK
    
adf_pipeline:
  - platform: i2s_audio
    type: audio_out
    id: adf_i2s_out
    i2s_audio_id: i2s_out
    i2s_dout_pin: GPIO33 # PCM5102 DIN

  - platform: i2s_audio
    type: audio_in
    id: adf_i2s_in
    i2s_audio_id: i2s_in
    i2s_din_pin: GPIO13 #INMP441 SD
    pdm: false
    channel: right
    sample_rate: 16000
    bits_per_sample: 32bit

microphone:
  - platform: adf_pipeline
    id: adf_microphone
    gain_log2: 3
    keep_pipeline_alive: true
    pipeline:
      - adf_i2s_in
      - self

media_player:
  - platform: adf_pipeline
    id: adf_media_player
    name: s3-dev_media_player
    keep_pipeline_alive: true
    internal: false
    pipeline:
      - self
      - adf_i2s_out

voice_assistant:
  microphone: adf_microphone
  use_wake_word: false
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 3.0
  media_player: adf_media_player
  id: assist

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(assist).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(assist).set_use_wake_word(false);

@Battman2013
Copy link

Hi imonlinux, many thanks for sharing your config. Good to know, that the media_player component can be configured with the idf framework also. I have tried this with a wroom32 board using very short wires, but it makes noise almost suppressing the speech.
Earlier I had some success with S3 board, but using the arduino framework. The output sound was clear, but the media_player function was not stable. And I could not figure out, how to get the mic work with the assist pipeline. Physicly the mic was OK (I tested it with a separate Arduino test program on the same board), but Home Assistant did not recognize it.
I plan to try your config on the S3 board, if possible, to see, how it performs...

@imonlinux
Copy link

Hey Battman2013, I have been hacking away at a config for the ESP32 S3 N16R8. I purchased some from Amz and the following config is working pretty well. I was troubleshooting the MAX98357A only working on the 3.3v pin when I ran across this thread:
https://forum.arduino.cc/t/chinese-esp32-s3-5v-pin-warning/1192758
Once I bridged the IN-OUT pads, the 5v pin started making 5v. Also, I'm only using the onboard LED for the light. Saving my 16bit rings for boards that don't have them, like the XIAO ESP32 S3. Still troubleshooting the MIC on that one.

Next up is local wake word.

  device_name: "esp32-s3-n16r8-media-assist"
  friendly_name: "Media Assistant"
  device_description: "ESP32 S3 N16R8"
  api_key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  esp_board: "esp32-s3-devkitc-1"
  framework_type: "esp-idf"
  din: "GPIO10"   # MAX98357A - DIN
  lrclk: "GPIO8" # MAX98357A - LRCLK
  bclk: "GPIO9"  # MAX98357A - BCLK
  sd: "GPIO4"    # INMP441 - SD
  ws: "GPIO5"    # INMP441 - WS
  sck: "GPIO6"   # INMP441 - SCK
  l_r: "right"   # INMP441 - L/R (GND = right / 3.3v = left)
  di: "GPIO48"   # WS2812 - DI
  # Phases of the Voice Assistant
  # IDLE: The voice assistant is ready to be triggered by a wake-word
  voice_assist_idle_phase_id: '1'
  # LISTENING: The voice assistant is ready to listen to a voice command (after being triggered by the wake word)
  voice_assist_listening_phase_id: '2'
  # THINKING: The voice assistant is currently processing the command
  voice_assist_thinking_phase_id: '3'
  # REPLYING: The voice assistant is replying to the command
  voice_assist_replying_phase_id: '4'
  # NOT_READY: The voice assistant is not ready
  voice_assist_not_ready_phase_id: '10'
  # ERROR: The voice assistant encountered an error
  voice_assist_error_phase_id: '11'
  # MUTED: The voice assistant is muted and will not reply to a wake-word
  voice_assist_muted_phase_id: '12'

external_components:
  - source:
      type: git
      url: https://github.com/gnumpi/esphome_audio
      ref: main
    components: [ adf_pipeline, i2s_audio ]

esphome:
  name: ${device_name}
  comment: ${device_description}
  friendly_name: ${friendly_name}
  min_version: 2024.1.0
  name_add_mac_suffix: false
  platformio_options:
    board_build.flash_mode: dio
    board_upload.maximum_size: 16777216
  on_boot:
    - priority: 600
      then:
        - light.turn_on:
            id: led_ring
            brightness: 70%
            effect: connecting

esp32:
  board: ${esp_board}
  variant: ESP32S3
  flash_size: 16MB
  framework:
    type: ${framework_type}
    version: recommended
    sdkconfig_options:
      # need to set a s3 compatible board for the adf-sdk to compile
      # board specific code is not used though
      CONFIG_ESP32_S3_BOX_BOARD: "y"
      CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM: "16"
      CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM: "512"
      CONFIG_TCPIP_RECVMBOX_SIZE: "512"

      CONFIG_TCP_SND_BUF_DEFAULT: "65535"
      CONFIG_TCP_WND_DEFAULT: "512000"
      CONFIG_TCP_RECVMBOX_SIZE: "512"

psram:
  mode: octal
  speed: 80MHz

wifi:
  enable_rrm: true
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  fast_connect: true

ota:
  password: !secret ota_password

api:
  encryption:
    key: ${api_key}
  on_client_connected:
    then:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - delay: 1s
            - voice_assistant.start_continuous:
            - delay: 1s
            - voice_assistant.stop:
            - delay: 2s
            - voice_assistant.start_continuous:
            - script.execute: reset_led
  on_client_disconnected:
    then:
      - light.turn_on:
          id: led_ring
          blue: 0%
          red: 100%
          green: 100%
          brightness: 50%
          effect: connecting

logger:

button:
  - platform: restart
    id: restart_btn
    name: "${friendly_name} REBOOT"

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: ${ws}
    i2s_bclk_pin: ${sck}
  - id: i2s_out
    i2s_lrclk_pin: ${lrclk}
    i2s_bclk_pin: ${bclk}

adf_pipeline:
  - platform: i2s_audio
    type: audio_in
    id: adf_i2s_in
    i2s_audio_id: i2s_in
    i2s_din_pin: ${sd}
    pdm: false
    channel: left
    sample_rate: 16000
    bits_per_sample: 32bit
  - platform: i2s_audio
    type: audio_out
    id: adf_i2s_out
    i2s_audio_id: i2s_out
    i2s_dout_pin: ${din}

microphone:
  - platform: adf_pipeline
    id: adf_microphone
    gain_log2: 3
    keep_pipeline_alive: false
    pipeline:
      - adf_i2s_in
      - self
      
media_player:
  - platform: adf_pipeline
    id: adf_media_player
    name: media_player
    keep_pipeline_alive: false
    internal: false
    pipeline:
      - self
      - adf_i2s_out

voice_assistant:
  id: voice_asst
  microphone: adf_microphone
  media_player: adf_media_player
  noise_suppression_level: 4
  auto_gain: 31dBFS
  volume_multiplier: 15
  use_wake_word: false
  on_listening:
    - light.turn_on:
        id: led_ring
        blue: 100%
        red: 0%
        green: 0%
        brightness: 100%
        effect: wakeword
  on_tts_start:
    - light.turn_on:
        id: led_ring
        blue: 0%
        red: 0%
        green: 100%
        brightness: 75%
        effect: pulse
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          media_player.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led_ring
        blue: 0%
        red: 100%
        green: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

script:
  - id: reset_led
    then:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - light.turn_on:
                id: led_ring
                blue: 100%
                red: 0%
                green: 0%
                brightness: 50%
                effect: none
          else:
            - light.turn_off: led_ring

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(voice_asst).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - script.execute: reset_led

light:
  - platform: esp32_rmt_led_strip
    id: led_ring
    name: "${friendly_name} Light"
    pin: ${di}
    num_leds: 16
    rmt_channel: 0
    rgb_order: GRB
    chipset: ws2812
    default_transition_length: 0s
    effects:
      - pulse:
          name: "Pulse"
          transition_length: 0.5s
          update_interval: 0.5s
      - addressable_twinkle:
          name: "Working"
          twinkle_probability: 5%
          progress_interval: 4ms
      - addressable_color_wipe:
          name: "Wakeword"
          colors:
            - red: 0%
              green: 50%
              blue: 0%
              num_leds: 12
          add_led_interval: 20ms
          reverse: false
      - addressable_color_wipe:
          name: "Connecting"
          colors:
            - red: 60%
              green: 60%
              blue: 60%
              num_leds: 12
            - red: 60%
              green: 60%
              blue: 0%
              num_leds: 12
          add_led_interval: 100ms
          reverse: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment