evan-burke

# Jupyter runs its own event loop, so trying to work with asyncio causes problems in Jupyter.
# Errors are most commonly:
# RuntimeError: This event loop is already running

## NOTE: as of IPython 7.0, released in late 2018, this may not be needed, per:
# https://blog.jupyter.org/ipython-7-0-async-repl-a35ce050f7f7
# However, await(x) can be used anywhere in Jupyter, but will error on a standalone script. 
# Use asyncio.run(x) or something for .py scripts as described in https://docs.python.org/3/library/asyncio-task.html

# Solution:

evan-burke

Installing Dremio 1.4 on Ubuntu 16

### NOTE: this is significantly out of date since I last edited it in Jan 2018.
# See the comments on the gist for suggested changes for more recent versions.


##Install links / references
https://www.dremio.com/tutorials/recommender-scikit-learn-dremio-postgres-mongodb/
https://www.dremio.com/tutorials/dremio-oracle-aws/
https://docs.dremio.com/deployment/standalone-tarball.html

evan-burke

# https://github.com/rthalley/dnspython
# pip install dnspython

# NOTE reverse DNS lookups on Docker seem to be failing with Docker versions between ~18.6 and ~19.03.8, as of 2020-05-13
# This generates errors like:
# `The DNS response does not contain an answer to the question: 9.8.7.6.in-addr.arpa. IN PTR`
# Workaround: start containers with external DNS where possible using the --dns flag in 'docker run'

import dns.resolver
import sys

evan-burke

""" This is pretty basic but makes it a little eaiser to use sqlite3 for common tasks.

    Example usage:
    db = sqlitelib(dbfile)
    result = db.execute("select * from mytable")
    
    or:
    with sqlitelib(dbfile) as db:
      result = db.execute("select * from  mytable")

evan-burke

# work in progress. I'm not 100% sure on distinction between legacy compat32 api and new API yet, so grain of salt and all.


import email

# __init__ is empty in email.mime so you need to import specific files/modules.
import email.mime.multipart

evan-burke

# Use Pandas & SQLAlchemy.
# https://stackoverflow.com/questions/23103962/how-to-write-dataframe-to-postgres-table

# Note this will create a new table; see the 'faster option' at the above link for a method using 'copy' to an existing table.
# However, 'copy' cannot do an upsert; that requires inserting to a temp table, then upserting form temp table to destination table.

# This will lack PKs and FKs and indexes, of course, so if used naively you may see data duplication.

# Also the df.to_sql command can do an append (but not upsert), using the if_exists param:
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html

evan-burke

# simple password generator

# https://docs.python.org/3/library/secrets.html#recipes-and-best-practices

import argparse
import secrets
import string

# todo: add args for specialchars etc
# specify min numbers of capital letters, digits, etc.