Skip to content

Instantly share code, notes, and snippets.

@EverythingSmartHome
Last active February 14, 2025 21:52
Show Gist options
  • Save EverythingSmartHome/055fbdde31a607ef9d695d5cac780e94 to your computer and use it in GitHub Desktop.
Save EverythingSmartHome/055fbdde31a607ef9d695d5cac780e94 to your computer and use it in GitHub Desktop.
ESP32 & ESPHome Voice Assistant
esphome:
name: esp32-mic-speaker
friendly_name: esp32-mic-speaker
on_boot:
- priority: -100
then:
- wait_until: api.connected
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
esp32:
board: esp32dev
framework:
type: esp-idf
version: recommended
# Enable logging
logger:
# Enable Home Assistant API
api:
ota:
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "Esp32-Mic-Speaker"
password: "9vYvAFzzPjuc"
i2s_audio:
i2s_lrclk_pin: GPIO27
i2s_bclk_pin: GPIO26
microphone:
- platform: i2s_audio
id: mic
adc_type: external
i2s_din_pin: GPIO13
pdm: false
speaker:
- platform: i2s_audio
id: big_speaker
dac_type: external
i2s_dout_pin: GPIO25
mode: mono
voice_assistant:
microphone: mic
use_wake_word: false
noise_suppression_level: 2
auto_gain: 31dBFS
volume_multiplier: 2.0
speaker: big_speaker
id: assist
switch:
- platform: template
name: Use wake word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
entity_category: config
on_turn_on:
- lambda: id(assist).set_use_wake_word(true);
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
on_turn_off:
- voice_assistant.stop
- lambda: id(assist).set_use_wake_word(false);
@indevor
Copy link

indevor commented Jan 18, 2025

I'm having success with this setup, but noticing that the INMP441/ESP32S3 doesn't seem to be multiplying the volume of the mic even when "volume_multiplier: 15.0" is uncommented. I've tried 4x, all the way to 128x and the audio files that are passed to HA sound the same (that is, very low). This affects HA's ability to correctly do STT. I find that if I manually multiply the recorded sample, it is not at all distorted and relatively high quality, so there seems to be plenty of headroom to increase the mic volume, I just can't make the setup actually do it.

Any ideas? This is really the final sticking point for me being able to use these regularly in the house.

I think it should be implemented in the domain: microphone, it was done in fork https://github.com/gnumpi/esphome_audio, gain_log2 setting, the recorded sound was louder. However, this is not present in nabu_microphone

@indevor
Copy link

indevor commented Jan 18, 2025

Maybe @formatBCE could look at and implement gain_log2 (microphone gain) if he had the time and desire, because we are plugging in his implementation (nabu_microphone). But I don't know how to mention him in this discussion) and how to make a function request in his github).

@formatBCE
Copy link

Maybe @formatBCE could look at and implement gain_log2 (microphone gain) if he had the time and desire, because we are plugging in his implementation (nabu_microphone). But I don't know how to mention him in this discussion) and how to make a function request in his github).

Guys, nabu_microphone isn't mine, it's done by Nabu Casa devs - I'm just using it for Koala with simple forked resampled version.
You can bump gain on nabu_microphone with amplify_shift: https://github.com/esphome/home-assistant-voice-pe/blob/dev/home-assistant-voice.yaml#L1414

@indevor
Copy link

indevor commented Jan 18, 2025

Maybe @formatBCE could look at and implement gain_log2 (microphone gain) if he had the time and desire, because we are plugging in his implementation (nabu_microphone). But I don't know how to mention him in this discussion) and how to make a function request in his github).

Guys, nabu_microphone isn't mine, it's done by Nabu Casa devs - I'm just using it for Koala with simple forked resampled version. You can bump gain on nabu_microphone with amplify_shift: https://github.com/esphome/home-assistant-voice-pe/blob/dev/home-assistant-voice.yaml#L1414

It works much better now, thank you!

@formatBCE
Copy link

@Wetzel402 @indevor you may use non-forked version since you have separate I2S buses for input and output. Just set 16000 sample rate for microphone.
My fork is resampling from 48kHz to 16kHz, because on Respeaker Lite board single I2S bus used for both speaker and microphone, and voice assistant requires 48kHz for speaker, and 16kHz for mic.

@indevor
Copy link

indevor commented Jan 18, 2025

Yes now everything works via github.com/esphome/voice-kit repository, thanks for the explanation.

@hollosipeti
Copy link

I have been trying to get a voice assistant setup on an esp32 for months now with the ability to also have the media player. Is this possible? I also would like to use the same pin for both speaker and media player

It is possible with the nabu media player...

It requires an ESP32S3

For my build I'm also using a MAX98357A with 2 Adafruit ICS43434. There should be enough GPIO to run everything on separate pins which is also necessary for proper operation.

Hello! Can I use this code with an INMP441 microphone?

@indevor
Copy link

indevor commented Jan 21, 2025

I have been trying to get a voice assistant setup on an esp32 for months now with the ability to also have the media player. Is this possible? I also would like to use the same pin for both speaker and media player

It is possible with the nabu media player...
It requires an ESP32S3
For my build I'm also using a MAX98357A with 2 Adafruit ICS43434. There should be enough GPIO to run everything on separate pins which is also necessary for proper operation.

Hello! Can I use this code with an INMP441 microphone?

Hi, the original code and the code that is published last from Wetzel402 works with INMP441. For the code from @Wetzel402 I needed two microphones and modify it a bit (remove parts I don't need).

@aryajuanda
Copy link

aryajuanda commented Jan 29, 2025

I have been trying to get a voice assistant setup on an esp32 for months now with the ability to also have the media player. Is this possible? I also would like to use the same pin for both speaker and media player

It is possible with the nabu media player...
It requires an ESP32S3
For my build I'm also using a MAX98357A with 2 Adafruit ICS43434. There should be enough GPIO to run everything on separate pins which is also necessary for proper operation.

Hello! Can I use this code with an INMP441 microphone?

Its work with 2 inmp441,
I ‘m using 2 inmp441, max98357, rotary encoder and 12 ring led ws2812b to replicate voice assistant pe
edit some code, use external from https://github.com/esphome/home-assistant-voice-pe
for components:
- media_player
- micro_wake_word
- microphone
- nabu
- voice_assistant
- nabu_microphone

add some code from https://github.com/esphome/home-assistant-voice-pe/blob/dev/home-assistant-voice.yaml

center button, multiple click, rotary, and led are working with few adjustment

IMG_2959

@hollosipeti
Copy link

Hello!

Thanks for the answer. I have a similar setup, with one microphone for now. esphome/esphome#7672 With these external sources. But with a bigger speaker. I have two problems with it. 1, If the music is loud, can't hear "OK, Nabu". 2, Also, the music is much louder than the TTS. Is there any way to adjust the proportions?
20250130_145658

@aryajuanda
Copy link

Hello!

Thanks for the answer. I have a similar setup, with one microphone for now. esphome/esphome#7672 With these external sources. But with a bigger speaker. I have two problems with it. 1, If the music is loud, can't hear "OK, Nabu". 2, Also, the music is much louder than the TTS. Is there any way to adjust the proportions? 20250130_145658

If you have a big speaker, you must put your mic a littlebit far from the speaker, when it detect wake word, the media player will enter ducking mode to lower the volume so you can talk to the assistant

@indevor
Copy link

indevor commented Jan 30, 2025

Hello!

Thanks for the answer. I have a similar setup, with one microphone for now. esphome/esphome#7672 With these external sources. But with a bigger speaker. I have two problems with it. 1, If the music is loud, can't hear "OK, Nabu". 2, Also, the music is much louder than the TTS. Is there any way to adjust the proportions? 20250130_145658

I don't think it would work adequately with a speaker of that size, as there is no chip to force voice recognition (voice\speech extraction) in this project, as you would need to shout louder than a speaker with an amplifier. If I were you, I would use bluetooth + amplifier + speaker in one part of the room and the voice assistant in another. Without combining the two. Or move the speaker away from the assistant on the wires.

@hollosipeti
Copy link

Ok, thanks! I'm just testing it. I'll make a box for it. And the speaker will be a little further away.

@aryajuanda
Copy link

IMG_2961

my 1st version of budget home assistant voice is finished..

thx for the idea, discussion and sharing, hope that all external component will be available in main esphome repo soon

component:

  • esp32-s3-wroom
  • 2x inmp441 mic
  • max98357 amp
  • 40 mm hifi speaker
  • YD-040 rotary encoder (center button + dial)
  • ws2812 12 bit ring LED

maybe i will create a github repo later after finishing my 3d print case design

@cosminhriscu
Copy link

cosminhriscu commented Feb 2, 2025

Hello, I tried the code from Wetzel402. The only thing that I modified is psram to octal because I have N16R8 module. Also I use MAX98357A and one INMP441. But ESP cannot connect to Home Assistant. It gives the error "can't connect to esp. please make sure your yaml file contains an 'api:' line." although I have an api line with an encryption key. Any ideas what I'm doing wrong? Or has anyone a good or an alternative yaml code? Including the media player component from nabu. Here is my code:
'

substitutions:
  # Phases of the Voice Assistant
  # The voice assistant is ready to be triggered by a wake word
  voice_assist_idle_phase_id: '1'
  # The voice assistant is waiting for a voice command (after being triggered by the wake word)
  voice_assist_waiting_for_command_phase_id: '2'
  # The voice assistant is listening for a voice command
  voice_assist_listening_for_command_phase_id: '3'
  # The voice assistant is currently processing the command
  ### voice_assist_thinking_phase_id: '4'
  # The voice assistant is replying to the command
  voice_assist_replying_phase_id: '5'
  # The voice assistant is not ready
  voice_assist_not_ready_phase_id: '10'
  # The voice assistant encountered an error
  voice_assist_error_phase_id: '11'

  device_name: "aaaaaaa"
  friendly_name: "aaaaaaa"
  device_description: "aaaaaaa"
  din: "GPIO11"   # MAX98357A - DIN
  lrclk_mic: "GPIO15" # MAX98357A & ICS43434 - LRC/LRCL
  lrclk_amp: "GPIO9" # MAX98357A & ICS43434 - LRC/LRCL
  bclk_mic: "GPIO6"  # MAX98357A & ICS43434 - BCLK
  bclk_amp: "GPIO10"  # MAX98357A & ICS43434 - BCLK
  dout: "GPIO7"  # ICS43434 - DOUT
  # ICS43434 - SEL low = left, high = right
  duck: "30" # db reduction for audio ducking
  vol: "90%" # volume level at boot
  pwr: "GPIO37" # charger output
  tablet_battery: "sensor.kitchen_tablet_battery_level"
  
external_components:
  - source:
      type: git
      url: https://github.com/esphome/voice-kit
      ref: dev
    components:
      - media_player
      - micro_wake_word
      - microphone
      - nabu
      - voice_assistant
    refresh: 0s
  - source:
      type: git
      url: https://github.com/formatBCE/home-assistant-voice-pe
      ref: 48kHz_mic_support
    components:
      - nabu_microphone
    refresh: 0s

esphome:
  name: ${device_name}
  friendly_name: ${friendly_name}
  name_add_mac_suffix: true
  min_version: 2024.12.2
  platformio_options:
    board_build.flash_mode: dio
  on_boot:
    priority: 375
    then:
      - media_player.volume_set: ${vol}
      - delay: 10min
      - if:
          condition:
            lambda: return id(init_in_progress);
          then:
            - lambda: id(init_in_progress) = false;

esp32:
  board: esp32-s3-devkitc-1
  variant: esp32s3
  flash_size: 16MB
  framework:
    type: esp-idf
    version: recommended
    sdkconfig_options:
      CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240: "y"
      CONFIG_ESP32S3_DATA_CACHE_64KB: "y"
      CONFIG_ESP32S3_DATA_CACHE_LINE_64B: "y"
      CONFIG_ESP32S3_INSTRUCTION_CACHE_32KB: "y"
      CONFIG_ESP32_S3_BOX_BOARD: "y"
      CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY: "y"

      CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP: "y"

      # Settings based on https://github.com/espressif/esp-adf/issues/297#issuecomment-783811702
      CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM: "16"
      CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM: "512"
      CONFIG_ESP32_WIFI_STATIC_TX_BUFFER: "y"
      CONFIG_ESP32_WIFI_TX_BUFFER_TYPE: "0"
      CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM: "8"
      CONFIG_ESP32_WIFI_CACHE_TX_BUFFER_NUM: "32"
      CONFIG_ESP32_WIFI_AMPDU_TX_ENABLED: "y"
      CONFIG_ESP32_WIFI_TX_BA_WIN: "16"
      CONFIG_ESP32_WIFI_AMPDU_RX_ENABLED: "y"
      CONFIG_ESP32_WIFI_RX_BA_WIN: "32"
      CONFIG_LWIP_MAX_ACTIVE_TCP: "16"
      CONFIG_LWIP_MAX_LISTENING_TCP: "16"
      CONFIG_TCP_MAXRTX: "12"
      CONFIG_TCP_SYNMAXRTX: "6"
      CONFIG_TCP_MSS: "1436"
      CONFIG_TCP_MSL: "60000"
      CONFIG_TCP_SND_BUF_DEFAULT: "65535"
      CONFIG_TCP_WND_DEFAULT: "65535"  # Adjusted from linked settings to avoid compilation error
      CONFIG_TCP_RECVMBOX_SIZE: "512"
      CONFIG_TCP_QUEUE_OOSEQ: "y"
      CONFIG_TCP_OVERSIZE_MSS: "y"
      CONFIG_LWIP_WND_SCALE: "y"
      CONFIG_TCP_RCV_SCALE: "3"
      CONFIG_LWIP_TCPIP_RECVMBOX_SIZE: "512"

      CONFIG_BT_ALLOCATION_FROM_SPIRAM_FIRST: "y"
      CONFIG_BT_BLE_DYNAMIC_ENV_MEMORY: "y"
  
psram:
  mode: octal # quad for N8R2 and octal for N16R8
  speed: 80MHz

globals:
  # Global initialization variable. Initialized to true and set to false once everything is connected. Only used to have a smooth "plugging" experience
  - id: init_in_progress
    type: bool
    restore_value: no
    initial_value: 'true'
  # Global variable tracking the phase of the voice assistant (defined above). Initialized to not_ready
  - id: voice_assistant_phase
    type: int
    restore_value: no
    initial_value: ${voice_assist_not_ready_phase_id}
  # Global variable storing the first active timer
  - id: first_active_timer
    type: voice_assistant::Timer
    restore_value: false
  # Global variable storing the timer finished TTS
  - id: timer_tts_str
    type: std::string
    restore_value: false

logger:
    
captive_portal:

web_server:

api:
  id: api_id

ota:
  - platform: esphome
    password: "36e011a812d79f51cb886ff8d171eda2"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  ap:
    ssid: "Esp32-Mic-Speaker"
    password: "9vYvAFzzPjuc"

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: ${lrclk_mic}
    i2s_bclk_pin: ${bclk_mic}
  - id: i2s_out
    i2s_lrclk_pin: ${lrclk_amp}
    i2s_bclk_pin: ${bclk_amp}

microphone:
 - platform: nabu_microphone
   i2s_din_pin: ${dout}
   adc_type: external
   pdm: false
   sample_rate: 48000
   bits_per_sample: 32bit
   i2s_audio_id: i2s_in
   channel_0:
     id: mic0
   channel_1:
     id: mic1

speaker:
  - platform: i2s_audio
    id: spk
    sample_rate: 48000
    i2s_dout_pin: ${din}
    bits_per_sample: 32bit
    i2s_audio_id: i2s_out
    dac_type: external
    channel: mono
    timeout: never
    buffer_duration: 100ms

media_player:
  - platform: nabu
    id: nabu_media_player
    name: Media Player
    internal: false
    speaker:
    sample_rate: 48000
    volume_increment: 0.05
    volume_min: 0.4
    volume_max: 1
    on_announcement:
      - nabu.set_ducking:
          decibel_reduction: ${duck}
          duration: 0.0s
    on_state:
      if:
        condition:
          and:
            - switch.is_off: timer_ringing
            - not:
                voice_assistant.is_running:
            - not:
                lambda: return id(nabu_media_player)->state == media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING;
        then:
          - nabu.set_ducking:
              decibel_reduction: 0
              duration: 1.0s

    files:
      - id: timer_finished_sound
        file: https://github.com/esphome/home-assistant-voice-pe/raw/dev/sounds/timer_finished.flac
      - id: wake_word_triggered_sound
        file: https://github.com/esphome/home-assistant-voice-pe/raw/dev/sounds/wake_word_triggered.flac
      - id: jack_connected_sound
        file: https://github.com/esphome/home-assistant-voice-pe/raw/dev/sounds/jack_connected.flac
      - id: jack_disconnected_sound
        file: https://github.com/esphome/home-assistant-voice-pe/raw/dev/sounds/jack_disconnected.flac

micro_wake_word:
  id: mww
  microphone: mic0
  models:
    - model: hey_jarvis
      id: hey_jarvis
    - model: https://github.com/kahrendt/microWakeWord/releases/download/stop/stop.json
      id: stop
      internal: true
  vad:
  on_wake_word_detected:
    # If a timer is ringing: Stop it, do not start the voice assistant (We can stop timer from voice!)
    - if:
        condition:
          switch.is_on: timer_ringing
        then:
          - switch.turn_off: timer_ringing
        # Start voice assistant, stop current announcement.
        else:
          - if:
              condition:
                lambda: return id(nabu_media_player)->state == media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING;
              then:
                lambda: |-
                  id(nabu_media_player)
                    ->make_call()
                    .set_command(media_player::MediaPlayerCommand::MEDIA_PLAYER_COMMAND_STOP)
                    .set_announcement(true)
                    .perform();
              else:
                - script.execute:
                    id: play_sound
                    priority: true
                    sound_file: !lambda return id(wake_word_triggered_sound);
                - delay: 500ms
                - voice_assistant.start:

voice_assistant:
  id: va
  microphone: mic1
  media_player: nabu_media_player
  micro_wake_word: mww
  noise_suppression_level: 1
  auto_gain: 31dBFS
  volume_multiplier: 4.0
  on_client_connected:
    - lambda: id(init_in_progress) = false;
    - script.execute:
        id: play_sound
        priority: true
        sound_file: !lambda return id(jack_connected_sound);
    - micro_wake_word.start:
    - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
  on_client_disconnected:
    - voice_assistant.stop:
    - script.execute:
        id: play_sound
        priority: true
        sound_file: !lambda return id(jack_disconnected_sound);
    - lambda: id(voice_assistant_phase) = ${voice_assist_not_ready_phase_id};
  on_error:
    - if:
        condition:
          and:
            - lambda: return !id(init_in_progress);
            - lambda: return code != "duplicate_wake_up_detected";
        then:
          - lambda: id(voice_assistant_phase) = ${voice_assist_error_phase_id};
  # When the voice assistant starts: Play a wake up sound, duck audio.
  on_start:
    - nabu.set_ducking:
        decibel_reduction: ${duck}   # Number of dB quieter; higher implies more quiet, 0 implies full volume
        duration: 0.0s          # The duration of the transition (default is 0)
  on_listening:
    - lambda: id(voice_assistant_phase) = ${voice_assist_waiting_for_command_phase_id};
  on_stt_vad_start:
    - lambda: id(voice_assistant_phase) = ${voice_assist_listening_for_command_phase_id};
  on_stt_vad_end:
    - lambda: id(voice_assistant_phase) = ${voice_assist_thinking_phase_id};
  on_tts_start:
    - lambda: id(voice_assistant_phase) = ${voice_assist_replying_phase_id};
    # Start a script that would potentially enable the stop word if the response is longer than a second
    - script.execute: activate_stop_word_if_tts_step_is_long
  # When the voice assistant ends ...
  on_end:
    - wait_until:
        not:
          voice_assistant.is_running:
    # Stop ducking audio.
    - nabu.set_ducking:
        decibel_reduction: 0   # 0 dB means no reduction
        duration: 1.0s
    # Stop the script that would potentially enable the stop word if the response is longer than a second
    - script.stop: activate_stop_word_if_tts_step_is_long
    # Disable the stop word (If the timer is not ringing)
    - if:
        condition:
          switch.is_off: timer_ringing
        then:
          - lambda: id(stop).disable();
    # If the end happened because of an error, let the error phase on for a second
    - if:
        condition:
          lambda: return id(voice_assistant_phase) == ${voice_assist_error_phase_id};
        then:
          - delay: 1s
    # Reset the voice assistant phase id and reset the LED animations.
    - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
  on_timer_finished:
    - switch.turn_on: timer_ringing
  
button:
  - platform: restart
    id: restart_btn
    name: "${friendly_name} REBOOT"

switch:
  # Internal switch to track when a timer is ringing on the device.
  - platform: template
    id: timer_ringing
    optimistic: true
    internal: true
    restore_mode: ALWAYS_OFF
    on_turn_off:
      # Disable stop wake word
      - lambda: id(stop).disable();
      # Stop any current annoucement (ie: stop the timer ring mid playback)
      - if:
          condition:
            lambda: return id(nabu_media_player)->state == media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING;
          then:
            lambda: |-
              id(nabu_media_player)
                ->make_call()
                .set_command(media_player::MediaPlayerCommand::MEDIA_PLAYER_COMMAND_STOP)
                .set_announcement(true)
                .perform();
      # Set back ducking ratio to zero
      - nabu.set_ducking:
          decibel_reduction: 0
          duration: 1.0s
    on_turn_on:
      # Duck audio
      - nabu.set_ducking:
          decibel_reduction: ${duck}
          duration: 0.0s
      # Enable stop wake word
      - lambda: id(stop).enable();
      # Ring timer
      - script.execute: ring_timer
      # If 15 minutes have passed and the timer is still ringing, stop it.
      - delay: 15min
      - switch.turn_off: timer_ringing
  # switch to charge tablet
  - platform: gpio
    name: ${device_name} charger
    pin: ${pwr}
    id: pwr
    icon: mdi:charging_station      

sensor:
  - platform: homeassistant
    entity_id: ${tablet_battery}
    id: battery
    internal: true
    filters:
      - heartbeat: 10s
    on_value:
      - if:
          condition:
              - lambda: "return id(battery).state < 30;" # '70' is the lower level, change if needed
          then:
            - switch.turn_on: pwr
          else:
            - if:
                condition:
                  - lambda: "return id(battery).state > 80 ;" # '80' is the upper level, change if needed
                then:
                  - switch.turn_off: pwr

script:
  # Script executed when the timer is ringing, to playback sounds.
  - id: ring_timer
    then:
      - script.execute:
          id: timer_tts
      - while:
          condition:
            switch.is_on: timer_ringing
          then: # this is required to play the output on a media player
            - script.execute:
                id: play_sound
                priority: true
                sound_file: !lambda return id(timer_finished_sound);
            - delay: 4s
            - homeassistant.service:
                service: assist_satellite.announce
                data:
                  entity_id: assist_satellite.assist_media_player_kitchen_assist_satellite
                  message: !lambda return id(timer_tts_str);
            - wait_until:
                lambda: |-
                  return id(nabu_media_player)->state == media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING;
            - wait_until:
                not:
                  lambda: |-
                    return id(nabu_media_player)->state == media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING;

  # Script executed when we want to play sounds on the device.
  - id: play_sound
    parameters:
      priority: bool
      sound_file: "media_player::MediaFile*"
    then:
      - lambda: |-
          if (priority) {
            id(nabu_media_player)
              ->make_call()
              .set_command(media_player::MediaPlayerCommand::MEDIA_PLAYER_COMMAND_STOP)
              .set_announcement(true)
              .perform();
          }
          if ( (id(nabu_media_player).state != media_player::MediaPlayerState::MEDIA_PLAYER_STATE_ANNOUNCING ) || priority) {
            id(nabu_media_player)
              ->make_call()
              .set_announcement(true)
              .set_local_media_file(sound_file)
              .perform();
          }

  # Script used to fetch the first active timer (Stored in global first_active_timer)
  - id: fetch_first_active_timer
    then:
      - lambda: |
          const auto timers = id(va).get_timers();
          auto output_timer = timers.begin()->second;
          for (auto &iterable_timer : timers) {
            if (iterable_timer.second.is_active && iterable_timer.second.seconds_left <= output_timer.seconds_left) {
              output_timer = iterable_timer.second;
            }
          }
          id(first_active_timer) = output_timer;
    
  # Script used to create TTS string for timer
  - id: timer_tts
    then:
      - script.execute:
          id: fetch_first_active_timer
      - lambda: |-
          std::string finished_timer_name = id(first_active_timer).name;
          int total_seconds = id(first_active_timer).total_seconds;
          // Variables for hours, minutes, and seconds
          int hours = total_seconds / 3600;
          int minutes = (total_seconds % 3600) / 60;
          int seconds = total_seconds % 60;
          std::string finished_timer_duration;
          std::string result;
          // If more than 1 hour
          if (hours > 0) {
            finished_timer_duration = std::to_string(hours) + " hour" + (hours > 1 ? "s" : "");
            if (minutes > 0) {
              finished_timer_duration += " " + std::to_string(minutes) + " minute" + (minutes > 1 ? "s" : "");
            }
            if (seconds > 0) {
              finished_timer_duration += " " + std::to_string(seconds) + " second" + (seconds > 1 ? "s" : "");
            }
          }
          // If less than 1 hour but more than 1 minute
          else if (minutes > 0) {
            finished_timer_duration = std::to_string(minutes) + " minute" + (minutes > 1 ? "s" : "");
            if (seconds > 0) {
              finished_timer_duration += " " + std::to_string(seconds) + " second" + (seconds > 1 ? "s" : "");
            }
          }
          // If less than 1 minute
          else {
            finished_timer_duration = std::to_string(seconds) + " second" + (seconds > 1 ? "s" : "");
          }
          // Construct the final message
          if (finished_timer_name.empty()) {
            result = finished_timer_duration + "timer finished";
          }
          else {
            result = finished_timer_name + "timer finished";
          }
          id(timer_tts_str) = result;

  # Script used activate the stop word if the TTS step is long.
  # Why is this wrapped on a script?
  #   Becasue we want to stop the sequence if the TTS step is faster than that.
  #   This allows us to prevent having the deactivation of the stop word before its own activation.
  - id: activate_stop_word_if_tts_step_is_long
    then:
      - delay: 1s
      # Enable stop wake word
      - lambda: id(stop).enable();`

@Wetzel402
Copy link

Try adding your encryption key like this:

api:
  encryption:
    key: !secret assist_kitchen_api

@cosminhriscu
Copy link

Try adding your encryption key like this:

api:
  encryption:
    key: !secret assist_kitchen_api

I did it directly in code, not with !secret...and it didn't work. Sould I try with !secret file?

@Wetzel402
Copy link

Try adding your encryption key like this:

api:
  encryption:
    key: !secret assist_kitchen_api

I did it directly in code, not with !secret...and it didn't work. Sould I try with !secret file?

That shouldn't mater. Can you give us your logs?

@cosminhriscu
Copy link

Try adding your encryption key like this:

api:
  encryption:
    key: !secret assist_kitchen_api

I did it directly in code, not with !secret...and it didn't work. Sould I try with !secret file?

That shouldn't mater. Can you give us your logs?

I figured out, I needed to delete the sensor for tablet battery. Now I'm adapting the code for only one microphone and also for adding a LED.
I see that you have 2 channels for the microphone. One channel I think is for the wakeword and the second one for the voice command. How can I use one microphone?

@Wetzel402
Copy link

If you only have one microphone I think you can just use the standard old mic example instead. I think the nabu mic requires two.

@cosminhriscu
Copy link

cosminhriscu commented Feb 7, 2025

Yes, for using nabu mic you must have 2 microphones. But how is the connection diagram for 2 mics? I have 2 INMP441 mics.

@indevor
Copy link

indevor commented Feb 7, 2025

Here I've made a schematic and assembled the device:
Schematic_esp32s3+nmp441+dac_2025-02-07

I want to warn about inflated expectations (maybe only mine).
As a device that stands on a table in a room where you have soft furniture, the floor is carpeted or padded and you communicate at a distance of 2 meters - it is acceptable. With the wake up word “ok nabu”, jarvis is in second place, alexa is practically deaf (too bad it's the only word that in theory isn't too cringe-worthy to say every day (who or what is nabu I don't know). To me as a resident of eastern Europe - everything except alexa is depressing (sorry).
The problems will start when you move 3-5 meters away, or when you put the device in a corridor (another room with no furniture) where there is echo and sound re-reflection is a big problem.
If someone is talking or making noise or knocking or playing music, it's a deaf device.

To summarize. Most likely I will bet on openwakeword - I think more complex neural networks will be able to filter noise, echo, suppress interference and so on. A large model on computer hardware will definitely win - without complex hardware of the device.. Also nobody canceled software DSP.
What is missing here:

  • microphone array (4-6) - direction detection and amplification. In this project there are two of them - 1 listens to the wake-up, the second listens to the command.
  • acoustic Echo Cancellation (AEC)
  • blind Source Separation (BSS)
  • noise Suppression (NS)
    Therefore, the project voice pe - has XMOS and aic3204, which at the hardware level on the fly trying to improve and increase the sound that you do not shout in an empty corridor 4 times from 5 meters.

@cosminhriscu
Copy link

Thanks. Those two capacitors on 3.3 V pin are mandatory? I connected two INMP441 mics, one with L/R pin to ground and the other one L/R to 3.3V pin. It worked for some hours in testing, after that one of them (I think the one with L/R pin to 3.3V) stopped working and now is dead. So now the wake word is working but no voice command is received. Also, the L1 is a resistor? I'm sorry I'm not so good at electronic diagrams.

@indevor
Copy link

indevor commented Feb 7, 2025

Thanks. Those two capacitors on 3.3 V pin are mandatory? I connected two INMP441 mics, one with L/R pin to ground and the other one L/R to 3.3V pin. It worked for some hours in testing, after that one of them (I think the one with L/R pin to 3.3V) stopped working and now is dead. So now the wake word is working but no voice command is received. Also, the L1 is a resistor? I'm sorry I'm not so good at electronic diagrams.

No it is not necessary, capacitors and inductors (l1, l2), I reinsured against dirty power supply. in your case most likely they are of poor quality or you mixed up and the voltage of 5v got to the microphones.

@AapoTahkola
Copy link

I though to reply to this with some of my findings. It has really taken me many many months to get all of this working flawlesly. I first tried using whisper on dell optiplex 3020 but as it turns out it is not fast enough to run. Luckily I recently found the perfect solution https://github.com/einToast/openai_stt_ha available from HACS create separate account for platform.openai.com and ditch 10 usd to it. It costs 0.006 usd per minute so even with heavy use it should not cost more than 1 usd per month to use. And since the account is not linked you get some privacy as well.

I find these settings useful:

voice_assistant:
  noise_suppression_level: 3.0
  auto_gain: 31dBFS
  volume_multiplier: 12.0

I for a long time had trouble with the assistant not responding and I realized I had to solder the dupont connectors as shown. Even after soldering I got some stutter on speaker and only after replacing the i2s dupont wires and resoldering they went away. Alot of these dupont wires have issues so be sure to check them.
I had some issues with wifi signal strength on esp32s3 N16R8 so I ended up cutting the PCB antenna trace with scissors and soldered slightly better antenna to scratched PCB traces on sides of the antenna. A bit of a nasty way of doing things but it improves wifi connection to acceptable level.

I have 4 scenes: coming_home, leaving_home, sleep, waking. Then for each I have automations with sentences like: Good morning, I am up, I'm up, Morning. Going out automation: Leaving home, I'm leaving home, I'm leaving, going out, I'm going out, I am leaving.

I used the tie-to-vcc gain setting of MAX98357, which is pretty good and not too loud if you need to watch out for neighbors at night.
Speaker is https://www.aliexpress.com/item/1005006524208930.html

I would say the this is not quite as good as Google Nest (not a native English speaker) but manageable it I speak out clearly enough and not too far away.

Spark as I like to call it.
3072-4096-max

4096-3072-max

@hollosipeti
Copy link

hollosipeti commented Feb 14, 2025

my 1st version of budget home assistant voice is finished..

thx for the idea, discussion and sharing, hope that all external component will be available in main esphome repo soon

component:

  • esp32-s3-wroom
  • 2x inmp441 mic
  • max98357 amp
  • 40 mm hifi speaker
  • YD-040 rotary encoder (center button + dial)
  • ws2812 12 bit ring LED

maybe i will create a github repo later after finishing my 3d print case design

Does the wake up work well for you? I noticed that if the media player is louder, it doesn't listen to the wake-up call. I got a smaller speaker. :) My components are mostly the same as yours. And otherwise, it would work fine.
component:

esp32-s3-wroom
2x inmp441 mic
max98357 amp
JBL satellite speaker (not big)
ws2812 16 bit ring LED

mic settings:
microphone:

  • platform: nabu_microphone
    i2s_din_pin: GPIO4
    adc_type: external
    pdm: false
    sample_rate: 48000
    bits_per_sample: 32bit
    i2s_audio_id: i2s_in
    channel_0:
    id: mic0
    amplify_shift: 2
    channel_1:
    id: mic1
    amplify_shift: 0

va:
voice_assistant:
id: va
microphone: mic1
media_player: nabu_media_player
micro_wake_word: mww
noise_suppression_level: 1
auto_gain: 31dBFS
volume_multiplier: 4.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment