Script to check the hashrate of ethereum worker on ethOs and reboot if the hashrate is 0 for 2 runs of this script. This script is for a single GPU, or rather it will only reboot if the overall hashrate reaches 0. It also uses the pushover service to push notifications for when the script triggers a reboot.

check_hashrate_reboot.py

import subprocess
import os
import re
import sys
import argparse
import httplib, urllib
import time

"""
# place this file at /home/ethos/check_hash_reboot.py
# The entries at the bottom of this comment block go into the crontab for the ethos user

# this uses pushover to notify you about reboots etc.
# replace <app_token> with your app token, eg. curl -s -F "token=as9dv8jaw4e9fasdjgoa2w3t98jawl4" ...
# replace <user_token> with your user token eg. ... -F "user=rtsfgjsrtjsrtj457ujtfgtwsvka" ...

## notifies you when EthOS is back up
@reboot curl -s -F "token=<app_token>" -F "user=<user_token>" -F "title=ethos back up" -F "message=Rebooted but now back up" https://api.pushover.net/1/messages.json

## runs this script every minute and passes in pushover tokens and pipes to log file 
 * * *   *    *   /usr/bin/python /home/ethos/check_hash_reboot.py -u <user_token> -a <app_token>  >> /home/ethos/check_hash_reboot.log
"""
# Send a message using the pushover notification service
def pushover_message(title, message, app_token, user_token):
	conn = httplib.HTTPSConnection("api.pushover.net:443")
	conn.request("POST", "/1/messages.json",
	  urllib.urlencode({
	    "token": app_token,                       # Insert app token here
	    "user": user_token,                       # Insert user token here
	    "title": title,                			  # Title of the message
	    "message": message     				      # Content of the message
	  }), { "Content-type": "application/x-www-form-urlencoded" })
	return conn.getresponse()

# parse the passed arguments
parser = argparse.ArgumentParser()

parser.add_argument(
	'-f',
	'--checkfilepath',
	dest='check_file_path', 
	help="path to store temporary file at if reboot criteria is met, but we need to wait until next time this script runs, check if that file exists, criteria are till met and then reboot",	
	default="/tmp/reboot_conditions_met_last_time.txt"
)

parser.add_argument('-a', '--pushover_app_token', dest='pushover_app_token',  help="app token for pushover service for push notifications on reboot")

parser.add_argument('-u', '--pushover_user_token', dest='pushover_user_token',  help="user token for pushover service for push notifications on reboot")

args = parser.parse_args()

# call the update command from EthOs, which outputs current status, including hashrate
update_data = subprocess.check_output(['/opt/ethos/bin/update'])

hash = None
miner_id = ''

# loop through the output of the update command, to parse the hashrate
for line in update_data.splitlines():
	if 'hash:  ' in line:
		hash_line = line
		hash_list = re.findall(r'\d+\.\d+', hash_line)

		# if we don't get a 2 decimal number, then it is probably crashed
		if len(hash_list) > 0:
			hash = float(hash_list[0])

	# store the hostname to add to the push notification
	elif 'hostname:   ' in line:
		hostname_list = re.findall(r'\w+', line)
		if len(hostname_list) > 1:
			miner_id = hostname_list[1]
# debugging output
print time.ctime()
print hash

# start doing stuff if the hash is non-existant of less than 10
if not hash or hash < 10:
	print "hash is zero"
	#criteria are met
	#check if file exists, meaning that conditions were met last time
	if os.path.isfile(args.check_file_path):
		print 'file here'
		os.remove(args.check_file_path)
		# send push notification if the tokens have been set
		if args.pushover_user_token and args.pushover_app_token:
			pushover_message(
				'Miner {} restarted'.format(miner_id),
				'Miner {} reached hash rate of 0'.format(miner_id),
				args.pushover_app_token,
				args.pushover_user_token
			)
		# reboot the system
		os.system("/opt/ethos/bin/r")
	else:
                print "making file {}".format(args.check_file_path)
		# create a file so that next time we know if has been at state for a while
		os.system('touch {}'.format(args.check_file_path))
else:
	print "Hash is good"
	# if the checkfile exists, remove it because the conditions are no longer met
	if os.path.isfile(args.check_file_path):
		os.remove(args.check_file_path)

This is a pain in the ass 👍

Im a doing it good?
"amod6szn8n718zxpx4vm768zafz2vw": app_token, # Insert app token here
"unt8dntiyncwoy74peza8wom4guffg": user_token, # Insert user token here
"title": title, # Title of the message
"message": message # Content of the message

Do i need to fill in more?

I am not getting any pushovers? i made a application and gave me a token and on my profile a user_token

No.
I have updated the file with a bit of a readme.

You put the file at /home/ethos/check_hash_reboot.py

As the ethos user edit the crontab with:
crontab -e

Add the two lines to the crontab, replacing the token placeholders with the pushover tokens.

@reboot curl -s -F "token=<app_token>" -F "user=<user_token>" -F "title=ethos back up" -F "message=Rebooted but now back up" https://api.pushover.net/1/messages.json
 * * *   *    *   /usr/bin/python /home/ethos/check_hash_reboot.py -u <user_token> -a <app_token>  >> /home/ethos/check_hash_reboot.log

From there it should be running.
I have had it reboot for me while I was at work and get back up and running again, so I know it works for me.

Hi hello. can you make a version of your script but with this conditions:

1. parse the fiel /var/run/ethos/miner_hashes.file (this file contains the hash rate of every GPU)
   example: all gpus Working:
   28.52 28.61 00.00 28.41 27.28 28.61
   some gpu crashed:
2. check if internet works (because it internet dont work the hash rate its 00.00)
3. if some hashrate of gpu report 00.00 and internet works for example makeing a ping to google
   execute the reboot command and use push over service to send a message

:) regards
Pablo

Hello,
THank you for putting this script together. I'm running it now. Shouldn't it be printing values time.ctime() and hash every minute? Nothing prints at my prompt.

Thank you very much, your script works for my rig. I did get push notification when it reboots.
My question is: instead of calling “bin/update”, we can call “show stats”? I got complains from TOO MANY UPDATES..... when view remotely from browser

Great work!

I had similar problems. I wrote a script to monitor hash rates of all GPUs in the same rig. If hash rate is too low, it will reboot ethOS.
https://gist.github.com/unchartedworks/6cf00262db3a5618ec244471b9d94fd2

@pabonchox I created a version that will meet your needs: https://gist.github.com/robbrown0/87968668118402ff016cbc96b27862cf

Can someone please help me with this ?
I dont know how to put this in mij ethos..

Please send me mail for help..

Timothydevisser@gmail.com