How to use the Home Assistant Toyota NA integration

README.md

The original repository of toyota-na was removed following a DMCA takedown notice, as detailed here. Consequently, the integration featured in this tutorial will not function for legal reasons. I regret to say I can no longer offer updates related to this tutorial. It has been a rewarding experience working on this project, and I am confident that you will discover alternative solutions to make it operational.

How to use the Home Assistant Toyota NA integration

• How to use the Home Assistant Toyota NA integration
  • Use IMAP sensor
    • Troubleshooting
  • Use your own domain
    • Use AWS
      • Register AWS account and setup billing
      • Create AWS S3 bucket
      • Create AWS lambda function
      • Configure AWS S3 to allow SES action to write and Lambda to read/write
      • Create the AWS SES email domain and verify
      • Setup AWS SES rule set
      • Verify the setup
      • Auto-forward your Toyota App OTP to AWS SES email
      • Setup the HA Toyota NA integrations
      • Troubleshooting AWS

The Toyota AP now requires 2FA to be enabled for all accounts. This makes the current Toyota NA integration in Home Assistant unusable. I've been playing with the OPT for a while and found two ways to retrieve the OTP in an automated way.

The IMAP solution is easy to start with, but not very stable as the verification code often get lost. The personal domain + AWS solution is more reliable, but requires a bit more setup.

If you have any suggestions or questions, please feel free to leave a comment.

Use IMAP sensor

This method applies if you do not have your own domain.

1. Setup the IMAP integration in HA using your Toyota app account email and password. The IMAP server is different for different email providers.

   https://www.home-assistant.io/integrations/imap/

   For Gmail you need to create a app password if you have 2FA enabled.

   Make sure the value of "sensor.imap_<your_email>" is the correct number of unread message in your email inbox, though we won't use this sensor directly.

2. Setup a Home assistant file notification service in your configurations.yaml. This will be used later when an email is received in the IMAP event.

   notify:
     - name: toyota_na_verification_code
       platform: file
       filename: "/config/www/verification_code/code.txt" # Or any other path under www. 

3. Create this file if it doesn't exist. Make sure you can access this file in your browser via https://<your HA URL>/local/verification_code/code.txt

4. Add an automation that triggers when an IMAP email is received. Then parse the content of the email to retrieve 6 digit verification and call the file notification service to store it last line of the file.

   Change the email to your Toyota app account email.

   ##========== Toyota NA verification code ==========##
   - id: toyota_na_verification_code_received
     alias: "Toyota NA verification code received"
     trigger:
       platform: event
       event_type: "imap_content"
     condition:
       condition: and
       conditions:
         - condition: template
           value_template: "{{ trigger.event.data['username'] == '<your toyota email>' }}"
         - condition: template
           value_template: "{{ 'verification code:' in trigger.event.data['text'] }}"
     action:
       - service: notify.toyota_na_verification_code
         data_template:
           message: >-
             {{ trigger.event.data["text"] | regex_findall_index("verification code:\s+([0-9]+)") }}

5. The ha-toyota-na integration has been modified to work with toyota-na-custom, a a customized version of toyota-na that supports OTP since I can't find where the original toyota-na repo is.

   Clone the following repo to your local. I'll make a PR to the original repo later.

   https://github.com/t0ny-peng/ha-toyota-na

   Copy the custom_components/toyota_na folder to /config/custom_components on your Home Assistant instance.

6. Restart Home assistant. Add the integration. You will be prompted to enter these information:

   1. Toyota app account email
   2. Toyota app account password
   3. OTP URL. https://<your HA URL>/local/verification_code/code.txt
   4. OTP Timeout. Typically 30 seconds is enough, but can be increased if the email is not received in time.

7. You should be able to see the integration is setup successfully. The integration can refresh the token when necessary.

Troubleshooting

If you cannot get the integration to work, try to trouble shoot with the following steps:

1. Check if you received the email in your inbox. If not, make sure the email account is correct.
2. Use any of your other email to send a test email to your Toyota app account email. Check if the IMAP sensor is updated.
3. Use the debugging tool in HA to listen to the imap_content event. Make sure the event fires when you receive the email.
4. Open the OTP URL and monitor if the new verification code is written to the file.
5. Check the log of the integration. It should print the error message if there's any.

Use your own domain

Sometimes the IMAP sensor cannot retrieve the verification code email in time, resulting in failure of setting up the integration. Thus it's better to use AWS Email service for a more reliable setup.

Prerequisite:

1. You have a domain that you can manage the DNS records. E.g., mydomain.com

   You can register a domain from any domain registrar. I use Cloudflare for my domain.

Use AWS

AWS is the most reliable way to receive an email and trigger a webhook via lambda. The workflow is:

Email send to your (sub) domain -> Received by AWS SES -> Email content saved to S3, AND Trigger a lambda function -> The verification is saved to a file on S3, accessible by HA

The setup is a bit complicated but once setup, it's very reliable.

Register AWS account and setup billing

You'll get $300 for the first year, but in reality the workflow will cost almost nothing, if any. A credit card is required for registration.

Create AWS S3 bucket

I'm close to us-west-2 so I choose this region. You can choose any region. This tutorial use the name my-s3-bucket as an example.

Go to AWS S3 us-west-2 and click Create bucket. Enter a unique name. ⚠️In the Block Public Access settings for this bucket section, unselect Block all public access and check the 4 checkboxes below as:

• ON
• ON
• OFF
• OFF

This is to make sure the verification can be accessed from HA Toyota integration.

Create AWS lambda function

Go to AWS Lambda us-west-2 and click Create function.

Make sure you select/input the following

• The type is Author from scratch
• ToyotaAppVerificationCode as the function name
• Python 3.11 as the Runtime
• Keep the rest unchanged

Then click Create function.

IIRC the lambda function by default has access to the Internet, unless it's connected to a VPC. But let me know if you run into timeout error later.

Go back to the lambda page and click the newly created and only one lambda function. In the coding section, paste the following code:

⚠️Don't forget to change the S3 bucket name⚠️

import json
import boto3
import http.client

def find_verification_code(content):
    # Define the prefix that appears before the verification code
    prefix = "verification code:"

    # Find the start of the verification code
    start_index = content.find(prefix)
    if start_index != -1:
        # Adjust the start index to the beginning of the actual code
        start_index += len(prefix)

        # Extract the verification code
        verification_code = content[start_index:start_index+7].strip()
        return verification_code
    else:
        return ""
        
def lambda_handler(event, context):
    message_id = event["Records"][0]["ses"]["mail"]["messageId"]
    print(f"messageId: {message_id}")
    
    aws_bucket = "⚠️⚠️YOUR AWS S3 BUCKET NAME⚠️⚠️"
    object_key = "code.txt" ## Using code.txt as example, but could be any name,
                            ## also change every occurrence of code.txt below

    # Create an S3 client
    s3 = boto3.client('s3')

    # Retrieve the email content from S3
    email_object = s3.get_object(Bucket=aws_bucket, Key=message_id)
    content = email_object["Body"].read().decode('utf-8')
    verification_code = find_verification_code(content)
    # Log the email content
    # print("content: ", content)
    print(f"verification code is: {verification_code}")

    ## If `verification_code` is empty, return error
    if verification_code == "":
        return {"statusCode": 400, "body": json.dumps("No verification code found")}


    # Create an S3 client
    s3 = boto3.client('s3')

    # Putting the verification code into the S3 bucket
    try:
        s3.put_object(Bucket=aws_bucket, Key=object_key, Body=verification_code)
        return {
            'statusCode': 200,
            'body': json.dumps('Verification code saved successfully')
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps('Error saving verification code: ' + str(e))
        }

Then press CMD + S(or CTRL + S on Windows) to save the code. Click Deploy to deploy the code.

Before leaving this page, go to the Configuration -> Permission tab, click the Role name link and save its ARN somewhere. Its format is like arn:aws:iam::<your AWS ID>:role/service-role/<Lambda name>-role-<xxxxxx>

Configure AWS S3 to allow SES action to write and Lambda to read/write

Go back to the AWS S3 page and click the my-s3-bucket bucket. Currently, it's not accessible by Home Assistant, SES or Lambda. Permission needs to be given so that,

• SES can save the email to the S3 bucket, under a unique key filename
• Lambda can read the email content, and save the verification to a file code.txt
• Home Assistant(or the public Internet) can read the verification code from the file code.txt

Go to the permission tab and scroll down to Bucket policy. By default it should be empty.

Paste the following policy. ⚠️⚠️Remember to change Lambda Role ARN and Bucket Name⚠️⚠️

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLambdaGetObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<Lambda Role ARN>"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::<⚠️⚠️Bucket name⚠️⚠️>/*"
        },
        {
            "Sid": "AllowSESPuts",
            "Effect": "Allow",
            "Principal": {
                "Service": "ses.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<⚠️⚠️Bucket name⚠️⚠️>/*",
            "Condition": {
                "StringEquals": {
                    "AWS:SourceAccount": "<⚠️⚠️Account ID⚠️⚠️>"
                }
            }
        },
        {
            "Sid": "PublicReadAccessForCodeTxt",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<⚠️⚠️Bucket name⚠️⚠️>/code.txt"
        }
    ]
}

Create the AWS SES email domain and verify

Go to the AWS SES page. Click Verified Identities on the left. It's empty by default. Now add your subdomain to the verified list, e.g., mail.aws.mydomain.com by clicking the Create Identity button.

Select Domain as Identity type and enter mail.aws.mydomain.com in the blank. Keep the rest unchanged and click Create Identity. The identity is created, but needs to be verified.

In the next page, you'll see some DNS records to set in the Publish DNS records tab. Go to your domain DNS management page and add these records.

In addition, very important, add a MX record, with mail.aws as the Name, inbound-smtp.us-west-2.amazonaws.com as the value and 10 as the priority. This ensures that all the emails sent to *@mail.aws.mydomain.com will be handled by AWS SES.

Wait for several minutes for the DNS records to propagate, and this identity will be verified by AWS.

Setup AWS SES rule set

With the domain verified and MX record added, AWS SES can now receive emails. However, rule set is needed to inform AWS SES what to do with the received emails.

Go to the Email Receiving page and create a rule set. Enter ToyotaOTPRuleSet as the name, then the empty rule set is created. Click Create Rule, give it a name, uncheck Spam and virus scanning, then click Next.

In the recipient conditions page, add an email address toyota@mail.aws.domain.com. This tells AWS SES to only handle emails sent to this address, and this email will become your Toyota app account email(see below). Click Next.

In the Actions page, two actions are needed.

1. Deliver to S3 bucket, then choose the S3 bucket created above.
2. Invoke AWS Lambda function, choose the lambda function created above.

Upon finalizing adding the rule, you'll be prompted with a question whether to add permission to allow this rule to invoke Lambda function. Click Add Permission.

Go back to the Email receiving page and click the rule set ToyotaOTPRuleSet. Click Set As Active button to activate the rule set.

Verify the setup

Use any email account to send a test email to toyota@mail.aws.domain.com with the content verification code: 123456. Then download the code.txt file using the following link:

curl https://s3.<your-aws-region>.amazonaws.com/<my-s3-bucket>/code.txt

The file should contain the 6 digit verification code.

If not, refer to the following troubleshooting section.

Auto-forward your Toyota App OTP to AWS SES email

E.g., if your Toyota app account email is john.doe@gmail.com, we need it to auto forward the Toyota App OTP email to the AWS SES trigger email address toyota@mail.aws.domain.com.

Ideally, you could use toyota@mail.aws.domain.com as the Toyota app account email, but toyota seems having trouble sending verification email to that account. Therefore I forward the email from Gmail to AWS SES, but you are welcome to try it.

Go to Gmail settings, Forwarding and POP/IMAP tab, click Add a forwarding button and enter toyota@mail.aws.domain.com. After a while, you should see a new file in the AWS S3 bucket. Download it and open it with any text editor.

You should see a link in the email content to approve this forward. Open that link in browser and approve the forward.

Then go back to Gmail, add a search filter with:

1. From being donotreply@toyotaconnectedservices.com
2. Has the words being verification code:

Click continue or create filter, select Forward it to and use the toyota@mail.aws.domain.com as target.

Setup the HA Toyota NA integrations

Follow steps 5 and 6 in the Use IMAP sensor section to install the customized integration. ⚠️Remember to use the following link in the OTP URL field:

https://s3.<your-aws-region>.amazonaws.com/<my-s3-bucket>/code.txt

If everything is setup correctly, you should be able to see the integration initialization finishes successfully and some new entities are created. The integration will automatically refresh the token when necessary, and resort to OTP when the token is expired, e.g., if Home Assistant is restarted.

Troubleshooting AWS

If you cannot get the integration to work, try to trouble shoot with the following steps:

1. Verify that the CNAME, TXT(if any) and MX records of your domain are set correctly.

2. Verify that the identity status is "Verified" in the AWS SES page.

3. Make sure the AWS S3 bucket permission is set correctly without typo, or wrong name of lambda role.

4. Use any email to send a test email to toyota@mail.aws.domain.com, and use the Cloudwatch log to check if the lambda function is triggered.

   You can find the Cloudwatch log in Lambda function -> Monitoring -> View CloudWatch logs button.

5. Download the file in AWS S3, and open it with any text editor to make sure the content is an email content.

6. If you see the code.txt file created in S3 bucket page, but the link is not downloadable from, e.g., shell or browser, check the bucket permission and the policy.

7. If the Cloudwatch log of Lambda function reports 403 as return code, this is likely that your Home Assistant is proxied by Cloudflare and the Bot Fight Mode is turned on. Try disable it.

Hi Tony, It is just a local install on my Windows computer, running in a Linux VM. So I was using my .local address http://192.168.xx.xx:8123/local/ verification_code/code.txt I am sure I copied everything else exactly as you wrote it.

…

On Mon, Dec 11, 2023 at 10:43 PM Tony Peng ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ @marksy1968 <https://github.com/marksy1968> if you didn’t configure HTTPS, then just use “http” in the url. The point is, make sure that URL is accessible from your browser or terminal. Try not use any “device.local” domain as they might not work properly inside the HA container. Try use this: http://<your domain.com>:<PORT>/…….. — Reply to this email directly, view it on GitHub <https://gist.github.com/t0ny-peng/f85b0a0dc3d46246ae17a9d7215e39da#gistcomment-4789901> or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDZEGABNGV3J7N3CS6K4WBTYI7HHFBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTENRWGA4DIMRQU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

-- Gary

@marksy1968 If you look at the config directory, did you see a www directory? The code.txt file should have this path: /config/www/verification_code/code.txt. Try manually create it.

You may also need to enable the HTTP platform by adding http: to your configuration.yaml file.

yes I do, it is www/community, not sure if it should be that. How would I enable http in config.yaml? I apologize, this is sort of new to me. I have attached 3 screen shots. Thanks.

…

On Mon, Dec 11, 2023 at 11:12 PM Tony Peng ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ @marksy1968 <https://github.com/marksy1968> If you look at the config directory, did you see a www directory? The code.txt file should have this path: /config/www/verification_code/code.txt. Try manually create it. You may also need to enable the HTTP platform by adding http: to your configuration.yaml file. — Reply to this email directly, view it on GitHub <https://gist.github.com/t0ny-peng/f85b0a0dc3d46246ae17a9d7215e39da#gistcomment-4789918> or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDZEGACEZDAQLADFPY7EAQLYI7KRVBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTENRWGA4DIMRQU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

-- Gary

I'm not seeing screenshots in your email reply. Maybe it's swollen by Github. You just need to add http: in the configuration.yaml file, and restart Home Assistant by going to Settings -> System -> Top right corner and then Restart Home Assistant

okay, I know I am being very basic here (and I apologize for that), but I added http: to my config.yaml checked and restarted. In my browser I now type:

http://192.168.XX.XX:8123/local/verifiation_code/code.txt and it says 404 file not found.

I have gone over your directions a couple of times.

• under www I have created a folder called verefication_code and inside that I have a file called code.txt. That file is empty.
• when i look at the state of sensor.imap it says 71 emails, although my actual inbox only has 51 emails.
• I added the notify section to config.yaml
• I added the automation to the automation.yaml (in the value_template area I put my email in there. Should username be a specific value? (I have left that as username)

I really appreciate you taking the time to help me with this.

Is your nginx running in the same container as home assistant, or is it running in another container or on the host?

it's running in a separate container

I'm wondering if this is the DNS issue. Maybe try use the IP where nginx runs. And, watch the log of Home assistant. Might find some hints there.

I'm able to resolve the local domain from within the home assistant and the error persists when using the IP address as suggested

As I played with both method, I find that the IMAP solution is very unstable. It fails to retrieve the verification probably once every 5 times. Therefore I strongly recommend using your own domain and AWS.

@t0ny-peng keep trying to troubleshoot this, but no luck.

from my review of the logs, I can see the request/response sequence where it authenticates and then reaches the OTP callback response:

"callbacks": [{"type": "TextOutputCallback", "output": [{"name": "message", "value": "Enter OTP received on your Phone/Email"}

at this point there's a warning about the integration doing a blocking sleep call:

2023-12-18 21:43:36.905 WARNING (MainThread) [homeassistant.util.async_] Detected blocking call to sleep inside the event loop by custom integration 'toyota_na' at custom_components/toyota_na/config_flow.py, line 34: await client.auth.login(user_input["username"], user_input["password"], user_input["otp_url"], user_input["otp_timeout"]), please create a bug report at https://github.com/widewing/ha-toyota-na/issues

I can see the new OTP code is retrieved from IMAP and updated in the code.txt file, however it looks like setting an OTP Timeout >~15 seconds results in an "Unexpected error"

Since the automation to detect the new verification email takes longer than 15 seconds, it always fails.

I was able to work around it by setting the timeout to 15 seconds and then manually writing the code I received to the code.txt file, the integration retrieved the value from otp_url and the integration setup was successful

@t0ny-peng Just adding to the chorus of appreciation for all the work you put into this. I followed your AWS guide and got everything working in HA again!

One note -- you should update this snippet from the Setup the HA Toyota NA integrations section at the end of the AWS guide:

Follow steps 4 and 5 in the Use IMAP sensor section to install the customized integration.

As written, it's currently steps 5 and 6 users need to follow :)

Losing access to connected services in HA was disappointing, and you saved the day. Many thanks!!

@dannyk81 Glad it worked for you. The blocking call is just a warning and doesn't actually block the other tasks. I had to use a longer timeout with IMAP since it's not very stable.

@brystmar Thanks for pointing out. I'll fix that.

@marksy1968 Sorry for the late reply. In your case, seems that the automation isn't working properly. Try removing these lines from the automation and see if any email containing verification code: will trigger the imap_content event

      - condition: template
        value_template: "{{ trigger.event.data['username'] == '<your toyota email>' }}"

I disagree that it is not blocking. Whenever I attempt to configure the integration, the supervisor times out and kills Home Assistant.

24-01-02 18:15:02 ERROR (MainThread) [supervisor.homeassistant.api] Error on call http://172.30.32.1:8123/api/core/state: 
24-01-02 18:15:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-01-02 18:15:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-01-02 18:15:36 ERROR (MainThread) [supervisor.homeassistant.api] Error on call http://172.30.32.1:8123/api/core/state: 
24-01-02 18:16:08 ERROR (MainThread) [supervisor.homeassistant.api] Error on call http://172.30.32.1:8123/api/core/state: 
24-01-02 18:16:19 ERROR (MainThread) [supervisor.homeassistant.api] Error on call http://172.30.32.1:8123/api/core/state: 

During that time, I am unable to access home assistant in any capacity, and eventually the supervisor restarts HA.

@jhenkens That might be. I'm using the official HAOS VM image and didn't suffer from this problem. Maybe one can change the sleep to a non-blocking thread? I'm not familiar with HA integration development, thus didn't know how to notify HA when the verification code is retrieved.

With that being said, I find that 5 seconds timeout for AWS method is stable enough.

I am using the IMAP method and my HA instance keeps crashing/rebooting after I submit my credentials. I believe I am having the same issue as @jhenkens. I am running on a RPi4. I have tried adjusting my timeout time to 30, 60, 120, and 300 with no effect. I have also checked to see if the code.txt file is reachable and it is. The codes update there correctly too (unless it crashes, then the last code isn't used).

On a separate note: Is there a way to set up the integration by only using the service call? I appreciate the IMAP auto-updating the OTP code, but it is an additional point of failure, I'm just trying to figure out where the error is happening and if I can isolate the IMAP issue from the integration actually adding the entities, it may be easier to debug. I also have very little familiarity with the IMAP integration so I may be botching it as well.

Here are my logs:
2024-01-16 11:28:42.709 WARNING (MainThread) [homeassistant.const] LENGTH_MILES was used from toyota_na, this is a deprecated constant which will be removed in HA Core 2025.1. Use UnitOfLength.MILES instead, please create a bug report at https://github.com/widewing/ha-toyota-na/issues
2024-01-16 11:28:42.727 WARNING (MainThread) [homeassistant.const] PRESSURE_PSI was used from toyota_na, this is a deprecated constant which will be removed in HA Core 2025.1. Use UnitOfPressure.PSI instead, please create a bug report at https://github.com/widewing/ha-toyota-na/issues
2024-01-16 11:28:42.767 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Prius Prime LE for toyota_na
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 406, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/toyota_na/init.py", line 35, in async_setup_entry
otp_url=entry.data["otp_url"],
~~~~~~~~~~^^^^^^^^^^^
KeyError: 'otp_url'
2024-01-16 11:29:26.708 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'message' is undefined when rendering '{{ message.body }}'
2024-01-16 11:29:26.709 ERROR (MainThread) [homeassistant.components.imap.coordinator] Error rendering IMAP custom template (Template<template=({{ message.body }}) renders=3>) for msguid 78194 failed with message: UndefinedError: 'message' is undefined
2024-01-16 11:29:27.104 ERROR (MainThread) [homeassistant.components.automation.toyota_grab_verification_code] Toyota Grab Verification Code: Error executing script. Error for call_service at pos 2: Error rendering data template: IndexError: list index out of range
2024-01-16 11:29:27.115 ERROR (MainThread) [homeassistant.components.automation.toyota_grab_verification_code] Error while executing automation automation.toyota_grab_verification_code: Error rendering data template: IndexError: list index out of range
2024-01-16 11:31:12.451 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'message' is undefined when rendering '{{ message.body }}'
2024-01-16 11:31:12.452 ERROR (MainThread) [homeassistant.components.imap.coordinator] Error rendering IMAP custom template (Template<template=({{ message.body }}) renders=4>) for msguid 78169 failed with message: UndefinedError: 'message' is undefined
2024-01-16 11:31:12.709 ERROR (MainThread) [homeassistant.components.automation.toyota_grab_verification_code] Toyota Grab Verification Code: Error executing script. Error for call_service at pos 2: Error rendering data template: IndexError: list index out of range
2024-01-16 11:31:12.719 ERROR (MainThread) [homeassistant.components.automation.toyota_grab_verification_code] Error while executing automation automation.toyota_grab_verification_code: Error rendering data template: IndexError: list index out of range
2024-01-16 11:32:23.302 WARNING (MainThread) [homeassistant.util.async_] Detected blocking call to sleep inside the event loop by custom integration 'toyota_na' at custom_components/toyota_na/config_flow.py, line 34: await client.auth.login(user_input["username"], user_input["password"], user_input["otp_url"], user_input["otp_timeout"]), please create a bug report at https://github.com/widewing/ha-toyota-na/issues
2024-01-16 11:37:27.187 WARNING (MainThread) [hass_nabucasa.iot] Connection closed: Cannot write to closing transport

• 1. Check if you received the email in your inbox. If not, make sure the email account is correct.
• 2. Use any of your other email to send a test email to your Toyota app account email. Check if the IMAP sensor is updated.
• 3. Use the debugging tool in HA to listen to the imap_content event. Make sure the event fires when you receive the email. (I send a notification of the email's body to my phone and it never loads the full message as text, I assume it's whitespace and formatting)
• 4. Open the OTP URL and monitor if the new verification code is written to the file. (sometimes, if it doesn't crash)
• 5. Check the log of the integration. It should print the error message if there's any.

I've been running into the same issue. I run HA in a VM on unRAID and when I click submit to reauthenticate, the VM crashes followed by my whole unRAID server. I use email sensor with a 120 second timeout. It worked fine until some recent HA update.

t0ny's fork of toyota-na, the base library used, has a non-async "sleep" call on line 134 of auth.py. Sadly, the original repository that t0ny forked is subject to DMCA takedown because it allegedly has stolen code from Toyota. Which is the reason I'm guessing t0ny didn't put his change on Github. Both the original repository and t0ny's modification are available via PyPi, and since Python is an interpreted language, we can still see the source code.

I think the backing library could be changed to do the following:

1. In async def authorize(self, username, password): we should immediately attempt to load the current "codes" document from the server before trying to authorize, and cache the last-line contents. We should then continue as normal in the method.

    async def authorize(self, username, password):
        previous_otp = await self.request_otp()
...
                            cb["input"][0]["value"] = await self.request_otp(previous_value=previous_otp)

3. In request_otp, we should check the code document every X seconds (5?) in a loop, up to timeout, and return once we get a new code. As it is currently, _otp_timeout is not a timeout, it is a constant delay.

    async def request_otp(self, previous_value=None):
        start = time.time()
        while (time.time() - start) < self._otp_timeout:
            async with aiohttp.ClientSession() as session:
                async with session.get(self._otp_url) as resp:
                    if resp.status != 200:
                        raise LoginError()
                    otp = self._extract_otp(await resp.text())
                    if previous_value is None or otp != previous_value:
                        return otp
            await asyncio.sleep(5)
        raise TimeoutError("Failed to get OTP Code")

This should enable the OTP to happen as fast as HA receives the message (with up to 5-seconds added), as well as not-block since its using asyncio instead of time.sleep.

I haven't tested this yet, just proposing.

Confirmed via monkey-patch in config_flow.py. The integration succeeded in connecting and I have all entities available.

    async def async_get_entry_data(self, user_input, errors):
        async def request_otp(self, cache_value=False):
            import time, aiohttp, asyncio
            start = time.time()
            while (time.time() - start) < self._otp_timeout:
                async with aiohttp.ClientSession() as session:
                    async with session.get(self._otp_url) as resp:
                        if resp.status != 200:
                            raise Exception("Failed to login")
                        otp = self._extract_otp(await resp.text())
                        if cache_value:
                            self.cached_otp = otp
                            return None
                        elif otp != self.cached_otp:
                            return otp
                await asyncio.sleep(5)
            raise TimeoutError("Failed to get OTP Code")
            
        async def login(self, username, password, otp_url, otp_timeout):
            self._otp_url = otp_url
            self._otp_timeout = otp_timeout
            await self.request_otp(cache_value=True)
            authorization_code = await self.authorize(username, password)
            await self.request_tokens(authorization_code)

        try:
            client = ToyotaOneClient()
            client.auth.request_otp = request_otp.__get__(client.auth, ToyotaOneAuth)
            client.auth.login = login.__get__(client.auth, ToyotaOneAuth)
            await client.auth.login(user_input["username"], user_input["password"], user_input["otp_url"], user_input["otp_timeout"])

@brylee123 I honestly haven't seen that error before. Need to spend more time this weekend to check. Btw could you elaborate on what "system call" you want to use to replace IMAP? The fundamental problem that needs to be solved here is how to propagate the OTP to the integration setup process, which I'm not very familiar. My best try now is to add a sleep and it's been working reliable for me, using the AWS method. Maybe give it a try?

@ds-sebastian That's strange. Normally a crash in an integration will not bring down the HAOS with it, let alone the entire unRAID server. I had no idea what's wrong..

@jhenkens Yes I added a sleep to give the user enough time to enter password. I'm not an expert on HA integration development.

A "HOWEVER" UPDATE

Even though I strongly do not believe the original code in the original repo(https://github.com/toyotha/toyota-na) was "stolen" from inside, especially after reading the implementations, I have to remove my Pypi fork as well to avoid any legal risk. Sorry guys. If you still have the copy, use it with caution and do not redistribute.

I meant to authenticate within the service call where we can enter a 6 digit verification code and it would update code.txt. I messed up and had to reinstall the repo. And now it cannot find the toyota-na-custom library. Any pointers on where I can go to resolve this? Thanks!

Unfortunately, that library was taken down due to a DMCA request by Toyota. Remember this and vote with your wallet next time you’re in the market for a car.

…

On Sun, Jan 21, 2024 at 19:06 Barry Lee ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ I meant to authenticate within the service call where we can enter a 6 digit verification code and it would update code.txt. I messed up and had to reinstall the repo. And now it cannot find the toyota-na-custom library. Any pointers on where I can go to resolve this? Thanks! — Reply to this email directly, view it on GitHub <https://gist.github.com/t0ny-peng/f85b0a0dc3d46246ae17a9d7215e39da#gistcomment-4842913> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGGIGQWIOKEISJHX2VGHHI3YPXJSFBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTENRWGA4DIMRQU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

Yeah :( I figured.... well it was fun while it lasted! Time to disconnect from Toyota Connected Services. Was the library just a modified version of https://pypi.org/project/toyota-na/ ?

@brylee123 Yes it is. Some logics were added to retrieve the OTP from user provided URL.