Skip to content

Instantly share code, notes, and snippets.

@kylejohnson

kylejohnson/ecowater.py

Last active Aug 25, 2020
Embed
What would you like to do?
#!/usr/bin/python3
import requests
import json
import re
import paho.mqtt.client as mqtt
import paho.mqtt.publish as publish
# Regex to match the hidden input on the initial log in page
request_validation_re = re.compile(r'<input name="__RequestVerificationToken" type="hidden" value="([^"]*)" />')
# The serial number of your ecowater device
dsn = { "dsn": 'serialnumber' }
# The initial form data
payload = {
"Email" : "username",
"Password" : "password",
"Remember" : 'false'
}
# The headers needed for the JSON request
headers = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language' : 'en-US,en;q=0.5',
'X-Requested-With': 'XMLHttpRequest',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0'
}
# MQTT server details
mqtt_host = '127.0.0.1'
mqtt_port = 1883
mqtt_user = 'username'
mqtt_pass = 'password'
def querySite():
with requests.Session() as s:
# Initial GET request
try:
g = s.get('https://www.wifi.ecowater.com/Site/Login')
except requests.exceptions.RequestException as e:
print(f"Problem connecting to ecowater.com: {e}")
raise
# Grab the token from the hidden input
tokens = request_validation_re.findall(g.text)
payload['__RequestVerificationToken'] = tokens[0]
# Log in to the site
try:
login = s.post('https://www.wifi.ecowater.com/Site/Login', data=payload)
except requests.exceptions.RequestException as e:
print(f"Problem logging in to ecowater.com: {e}")
raise
# Add the correct Referer header
headers['Referer'] = login.url + '/' + dsn['dsn']
# Query the JSON endpoint for the data that we actually want
try:
data = s.post('https://www.wifi.ecowater.com/Dashboard/UpdateFrequentData', data=dsn, headers=headers)
except requests.exceptions.RequestException as e:
print(f"Problem getting JSON from ecowater.com: {e}")
raise
if data.status_code != 200:
print("Status code from ecowater.com was not 200")
raise ValueError("Status code from ecowater.com was not 200")
parseData(data.text)
def parseData(text):
# Load the data in to json
j = json.loads(text)
# Ensure at least one of the returned values makes sense (sanity check)
# All values were "0" once or twice.
assert j['water_flow'] >= 0, "Water flow does not appear to be a real number."
# Placeholder for each message
messages = []
# Extract the next recharge date, into something usable.
# False if we match 'Not Scheduled', else True
# Only sets this topic if we have the initial regex match.
nextRecharge_re = "device-info-nextRecharge'\)\.html\('(?P<nextRecharge>.*)'"
nextRecharge_result = re.search(nextRecharge_re, j['recharge'])
if nextRecharge_result:
msg = {
'topic': 'ecowater/rechargeTomorrow',
'payload': False if nextRecharge_result.group('nextRecharge') == 'Not Scheduled' else True
}
messages.append(msg)
# Delete the elements that we don't want
del j['out_of_salt']
del j['out_of_salt_days']
del j['water_units']
del j['time']
del j['recharge']
# Format each piece of json data in to a mqtt 'message'
for d in j:
msg = {
'topic': 'ecowater/' + d,
'payload': j[d]
}
messages.append(msg)
publishMessages(messages)
def publishMessages(messages):
# Publish the message, consisting of all of the json data
if len(messages) > 0:
publish.multiple(messages, hostname=mqtt_host, port=mqtt_port, auth={'username': mqtt_user, 'password': mqtt_pass})
else:
print('Messages is empty - nothing to publish.')
if __name__ == '__main__':
querySite()
@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented May 11, 2020

Great that you have found a way to extract data from Ecowater! Have you considered making this available an integration to Home Assistant - eg. via HACS?

@xundre

This comment has been minimized.

Copy link

@xundre xundre commented May 15, 2020

This is awesome, I was able to implement it , only problem is homeassistant complains about the "import" on the python script , how do you run the python script via homeassistant ?

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented May 15, 2020

Happy to hear it works for you!
I don’t use it directly in home assistant, I run it as a cron job, it sends the data to MQTT, and home assistant picks up the data from MQTT. You can see more details on my linked blog post.

@xundre

This comment has been minimized.

Copy link

@xundre xundre commented May 15, 2020

Yes , using the same flow of publishing to MQTT, I set up an MQTT docker just for this. Now I have 3 dockers homeassistant , MQTT and homebridge.

Not sure I can run cron in the homeassistant docker or the MQTT docker, I would hate to stand up another docker for cron jobs

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented May 17, 2020

is it possible to run this as an automation in hassio instead of a cron?

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented May 17, 2020

I'm attempting to run this as an automation but when I use call service I run into an error in my logs:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/python_script/init.py", line 205, in execute
exec(compiled.code, restricted_globals)
File "ecowater.py", line 3, in
ImportError: import not found

any idea what might cause this?

@xundre

This comment has been minimized.

Copy link

@xundre xundre commented May 18, 2020

@kchaney9 imports are not supported in homeassistant from what I read. I have not figured out yet how I want to run this, Cron was suggested , right not I just run it in my docker container for homeassistant using python ecowater.py

@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented May 18, 2020

My understanding is that you have to do it with either AppDaemon or create an integration.
I have not been using AppDaemon before - but managed to create an App which updates the ecowater sensors based on part of the code written by @kylejohnson. You can update sensors directly via AppDaemon so you don't need to use MQTT. I will be looking into creating a custom integration to be installed via HACS. I have not created integrations before and limited experience with Python so it might take a while...

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented May 18, 2020

Yeah, I have seen the AppDaemon apps in hacs but that's where my knowledge of it ends lol. If you would make an integration that would be awesome! I've been wanting to monitor the water usage in my home on HA but I didn't want to buy a sensor when I already have this service in my home.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented May 18, 2020

Hey everyone, I'm going to spend some time today trying to get this working in HomeAssistant directly. It isn't how I use the script - I just call it happily from cron - but I'll see what I can do, and report back.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented May 20, 2020

I don't want to go down the AppDaemon route and python_script: can't use imports, so it looks like the best option is for me to make this in to a custom integration.
In the mean time, it looks like it is fairly easy to add a cronjob to a docker host, so that should be able to hold some of you over.

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented May 20, 2020

If you make a custom integration that would be awesome! I have no experience with cron so I'm not sure where to begin with that but I'll try to look into it. I really appreciate you taking the time to work on this.

@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented May 29, 2020

I agree that a custom integration is the right way forward. I have in the mean time created a solution using Appdaemon.
The solution is based on a modified version of the script ecowater.py that @kylejohnson has written.

Below what I have done to get it running:

  1. Install AppDaemon Add-on via the Supervisor and start it
  2. Upload the modified version of ecowater.py and place in this folder: /config/appdaemon/apps (the file can just be uploaded via the File Editor integration in HA)
  3. Add the content of apps.yaml to /config/appdaemon/apps/apps.yaml
  4. Add Ecowater credentials and DSN to the secrets file /config/secrets.yaml
    ecowater_user:
    ecowater_password:
    ecowater_dsn:
  5. Define the schedule via the parameters below in apps.yaml. The schedule below is hourly 15:00 min past and daily at 23:59:00. The schedules can be disabled by removing the parameters from the file.
    dailyruntime: [23, 0, 59]
    hourlyruntime: [0, 15, 0]
  6. Restart AppDaemon in order to read the information from the secrets file

Currently there are two log statements (self.log...) in ecowater.py - so it is possible to see if the app is running and able to fetch data. The log output can be seen from the AppDaemon plugin (Log tab) via the Supervisor. AppDaemon will reload the code whenever one of the files are saved, so it is easy to check if it is working by just setting the hourlyruntime a minute later than current time and save the file.

I am new to Python, AppDaemon and github so the code is not perfect and might have to be done differently - but it works in my HA setup.

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented Jun 4, 2020

Thank you @blidegn for this! I was able to get it working, but I was wondering how I could get it to run much more frequently so I can accurately monitor current water usage.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented Jun 5, 2020

Unfortunately, water_flow is not very for monitoring water usage, at is only shows the water flow at the time that the softener reached out and uploaded to the API; water_today is much more useful in that regard.

I've also been attempting to connect a NodeMCU directly to the ecowater's flow sensor, but have had only inaccurate results thus far.

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented Jun 5, 2020

Well I was thinking if I could get it to do a callback every 10 seconds or so it would work for what I'm trying to do. I was able to adjust the hourly runtime for the next minute and ran the sink and it registered the flow because, as you said, it only shows the water flow for the moment it reached out to the API. So if it runs more frequently then it will give me a rough idea of the current water flow.

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented Jun 5, 2020

So I changed part of the modified ecowater.py from run_hourly to run_minutely so it's running once a minute which is much better. I tried to use run_every but I guess I couldn't get the formatting right (I'm very new to all of this) because I kept getting error messages with the time and interval part. I did some guessing and checking and googling but couldn't get it working so I've just settled for once a minute until I can get it working more often.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented Jun 8, 2020

I wouldn't recommend running it more than once a minute. For one, the script takes 20+ seconds to run and return any data. If you run it every 20 seconds, in this case, then it'll be overlapping on itself. Additionally, EcoWater could crack down on this, and block attempt to block us, if they think we're abusing their (poor excuse for an) API.

@kchaney9

This comment has been minimized.

Copy link

@kchaney9 kchaney9 commented Jun 8, 2020

Yeah, after living with it for a while, I think once a minute is enough for me anyways.

@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented Jun 8, 2020

@kylejohnson: I fully agree with you that it will not make sense to run the script more than once a minute due to the execution time and the fact that EcoWater might block access.

@kchaney9: I think that we need a proper API from EcoWater or somehow be able to poll the device directly in order to monitor the current water flow.

I have installed InfluxDB and Grafana in order to visualise the daily water usage via the sensor water_today and that works very well.

@quinten94b

This comment has been minimized.

Copy link

@quinten94b quinten94b commented Jul 28, 2020

@blidegn: I have your code running via appdaemon on HA, can you explain to me how I set it to pull data every 5 or 15 minutes instead every hour?

@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented Jul 30, 2020

@quinten94b: The code needs to be changed in order to pull data more often. Will see if I can get it done within the next week.

@blidegn

This comment has been minimized.

Copy link

@blidegn blidegn commented Aug 2, 2020

@quinten94b: A new parameter "runevery" has been added to ecowater.py and apps.yaml so it is possible to schedule the app to run based on number of minutes instead of hours.

@quinten94b

This comment has been minimized.

Copy link

@quinten94b quinten94b commented Aug 17, 2020

@quinten94b: A new parameter "runevery" has been added to ecowater.py and apps.yaml so it is possible to schedule the app to run based on number of minutes instead of hours.

Sorry for the late response: Thanks for adding the minute schedule. I'm heavily invested in Home Assistant but fairly new to the scripting part. I'm more of a PLC logic guy and I do understand the scripting but for the moment I'm not able to write it myself.

Do you know if there is a way to pull data so I know if my water softener is set to regenerate the next night or not? Some where in the website there is a code hidden with a string behind telling that the softener is set or not

<script type="text/javascript">$('#device-info-nextRecharge').html('Niet ingesteld');</script>
('Niet ingesteld' is Dutch for 'Not planned')

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented Aug 18, 2020

Do you know if there is a way to pull data so I know if my water softener is set to regenerate the next night or not? Some where in the website there is a code hidden with a string behind telling that the softener is set or not

I should be able to get this working.
What I see when there is not a recharge scheduled is $('#device-info-nextRecharge').html('Not Scheduled');. Do you know what is reported when there is a recharge scheduled?

@quinten94b

This comment has been minimized.

Copy link

@quinten94b quinten94b commented Aug 18, 2020

Do you know if there is a way to pull data so I know if my water softener is set to regenerate the next night or not? Some where in the website there is a code hidden with a string behind telling that the softener is set or not

I should be able to get this working.
What I see when there is not a recharge scheduled is $('#device-info-nextRecharge').html('Not Scheduled');. Do you know what is reported when there is a recharge scheduled?

If I remember correct it's 'Gepland' in Dutch or probably 'Scheduled' in English. But I'm not sure. I will be on the next regeneration in +-10 days.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented Aug 18, 2020

Turns out I don't need the answer to my above question. I added some code which essentially looks for Not Scheduled and sets rechargeTomorrow to false, else true.

Is that basically what you're looking for?

[
  {
    "topic": "ecowater/rechargeTomorrow",
    "payload": false
  },
  {
    "topic": "ecowater/online",
    "payload": true
  },
  {
    "topic": "ecowater/salt_level",
    "payload": 3
  },
  {
    "topic": "ecowater/salt_level_percent",
    "payload": 37.5
  },
  {
    "topic": "ecowater/water_today",
    "payload": 5
  },
  {
    "topic": "ecowater/water_avg",
    "payload": 77
  },
  {
    "topic": "ecowater/water_avail",
    "payload": 689
  },
  {
    "topic": "ecowater/water_flow",
    "payload": 0
  },
  {
    "topic": "ecowater/rechargeEnabled",
    "payload": true
  }
]
@quinten94b

This comment has been minimized.

Copy link

@quinten94b quinten94b commented Aug 18, 2020

Yes I'm looking to something like that. If I know it's not planned, I also know when it's planned.

@kylejohnson

This comment has been minimized.

Copy link
Owner Author

@kylejohnson kylejohnson commented Aug 18, 2020

I've updated the gist with my most recent version. I've restructured the code (it still produces the same results), and also added in the ecowater/rechargeTomorrow topic. From there, you could set up a mqtt binary sensor in home assistant with something like the following:

binary_sensor:
  - platform: mqtt
    name: "Water Softener Recharge Tomorrow"
    state_topic: "ecowater/rechargeTomorrow"
    payload_on: True
    payload_off: False

Hope this helps.

@BradleyFord

This comment has been minimized.

Copy link

@BradleyFord BradleyFord commented Aug 25, 2020

I'm just in the market for a water softener, if we can get this working in HA that would actually push me towards buying their product

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.