FHEM Auto FileLogConvert - A Python utility to convert all your logfiles one-by-one automatically with 98_FileLogConvert.pm

README.md

fhem_auto_import2DbLog

If you want to convert the logs of your FHEM installation from FileLog to DbLog, you may use the excellent 98_FileLogConvert.pm.

What's a bit inconvenient is that each logfile conversion needs to be started by hand. This is the issue, this Python tool solves: You provide it a list of logfiles and the conversion is done one by one automatically.

Why was this tool written in Python (not Perl as FHEM itself)? Because the author is just much more proficient and fast writing Python than Perl.

Usage:

$ ./fhem_auto_import2DbLog.py --help
usage: fhem_auto_import2DbLog.py [-h] [--host HOST] [--port PORT]
                                 [--file-log-convert FILE_LOG_CONVERT]
                                 [--retry-every-seconds RETRY_EVERY_SECONDS]
                                 [--wait-before-read WAIT_BEFORE_READ]
                                 file_list

positional arguments:
  file_list             A text file listing all the logfiles to convert, one
                        per line

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           The Telnet host, your FHEM instance is listening to.
  --port PORT           The Telnet port to connect to.
  --file-log-convert FILE_LOG_CONVERT
                        The FileLogConvert instance in FHEM that will be asked
                        to convert the logfiles.
  --retry-every-seconds RETRY_EVERY_SECONDS
                        How long should I wait until trying again to start the
                        next conversion?
  --wait-before-read WAIT_BEFORE_READ
                        How long shall the Telnet library wait before reading
                        any content back?

Example:

$ ./fhem_auto_import2DbLog.py ./list.txt --host 192.168.0.96

where list.txt should look like this (lines starting with a hashtag will be ignored):

FBDECT_19-2017.log
FHEMduino_PT2262_10_15-2016.log
#FHEMduino_PT2262_10_15-2017.log
BMP183_01-2017.log
speedtest-2017.log
IT_F0F0FF0FFF-2016.log

Resources

• FHEM SVN repository: 98_FileLogConvert.pm
• FHEM Wiki: https://wiki.fhem.de/wiki/FileLogConvert
• FHEM forum topic for FileLogConvert: https://forum.fhem.de/index.php/topic,66383.0.html

fhem_auto_import2DbLog.py

#!/usr/bin/env python

import sys, time
from telnetlib import Telnet

def import2DbLog(filenames, host='localhost', port=7072, retry_every_seconds=10, file_log_convert='LogConvert', wait_before_read=0.3):
    tn = Telnet(host, port)
    def write(string): tn.write(string.encode('utf-8'))
    def read(): time.sleep(wait_before_read); return tn.read_very_eager().decode('utf-8').strip()
    def status(message): sys.stderr.write(message + '\n'); sys.stderr.flush()
    write("list %s\n" % file_log_convert)
    test_output = read()
    if test_output.startswith('No device '):
        raise RuntimeError("Please specify the FileLogConvert instance. Tried '%s' but it doesn't exist" % file_log_convert)
    for filename in filenames:
        if not filename: continue
        while True:
            write("set %s import2DbLog %s\n" % (file_log_convert, filename))
            response = read()
            if response == '':
                status("%s ### starting to convert." % filename)
                break
            if 'has already been imported' in response:
                status("%s --- no need - already imported." % filename)
                break
            if 'Work already in progress' in response:
                status("%s waiting to start..." % filename)
                time.sleep(retry_every_seconds)
                continue
    write("exit\n")

def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('file_list', help='A text file listing all the logfiles to convert, one per line')
    parser.add_argument('--host', default='localhost', help='The Telnet host, your FHEM instance is listening to.')
    parser.add_argument('--port', type=int, default=7072, help='The Telnet port to connect to.')
    parser.add_argument('--file-log-convert', default='LogConvert', help='The FileLogConvert instance in FHEM that will be asked to convert the logfiles.')
    parser.add_argument('--retry-every-seconds', type=float, default=30, help='How long should I wait until trying again to start the next conversion?')
    parser.add_argument('--wait-before-read', type=float, default=1.0, help='How long shall the Telnet library wait before reading any content back?')
    args = parser.parse_args()
    filenames = []
    with open(args.file_list, 'r') as f:
        filenames += f.read().split('\n')
        filenames = [fn.strip() for fn in filenames]
        filenames = [fn for fn in filenames if fn != '']
        filenames = [fn for fn in filenames if not fn.startswith('#')]

    kwargs = {
        'host': args.host,
        'port': args.port,
        'file_log_convert': args.file_log_convert,
        'retry_every_seconds': args.retry_every_seconds,
        'wait_before_read': args.wait_before_read,
      }
    import2DbLog(filenames, **kwargs)

if __name__ == "__main__": main()

Hi!
Nice Code, helped me to save some time.
One issue: I inserted after line 18: time.sleep(4)
Because I run on an Raspberry and the respond was to slow and I got no response. So your code go on to the next file. With the wait between write and read I got save a response.
Thank you for sharing your code!
Best regards,
Burkhard