jimmytuc

psql: FATAL: Peer authentication failed for user “postgres” (or any user)

The connection failed because by default psql connects over UNIX sockets using peer authentication, that requires the current UNIX user to have the same user name as psql. So you will have to create the UNIX user postgres and then login as postgres or use sudo -u postgres psql database-name for accessing the database (and psql should not ask for a password).

If you cannot or do not want to create the UNIX user, like if you just want to connect to your database for ad hoc queries, forcing a socket connection using psql --host=localhost --dbname=database-name --username=postgres (as pointed out by @meyerson answer) will solve your immediate problem.

But if you intend to force password authentication over Unix sockets instead of the peer method, try changing the following pg_hba.conf* line:

from

jimmytuc

Popular ami image: ami-0da245167866d25b1

jimmytuc

def flatten_dataframe(df, columns, fillna_value='', preserve_index=False):
    if (columns is not None \
        and len(columns) > 0 \
        and not isinstance(columns, (list, tuple, np.ndarray, pd.Series))):
        columns = [columns]

    diff_columns = df.columns.difference(columns)

    # get column's series of length
    series_length = df[columns[0]].str.len()

jimmytuc

FROM python:3.7-alpine3.8

RUN apk add --no-cache \
            build-base \
            cmake \
            bash \
            jemalloc-dev \
            boost-dev \
            autoconf \
            zlib-dev \

jimmytuc

FROM python:3.7-alpine3.8

RUN apk add --no-cache \
            build-base \
            cmake \
            bash \
            jemalloc-dev \
            boost-dev \
            autoconf \
            zlib-dev \

jimmytuc

# import python core modules
import datetime
import logging

# import external modules
import pandas as pd
import requests

# import luigi modules
import luigi

jimmytuc

Collection of sample Elasticsearch queries

Use the Python client elasticsearch.

Connect to cluster (the client)

from elasticsearch import Elasticsearch

es_client = Elasticsearch() # local

jimmytuc

#/usr/bin/python3
import multiprocessing as mp
import os


def process_each_chunk(file, chunk_start, chunk_size):
    with open(file) as f:
        f.seek(chunk_start)
        lines = f.read(chunk_size).splitlines()
        for line in lines:

jimmytuc

Concurrency in JavaScript

Javascript is a programming language with a peculiar twist. Its event driven model means that nothing blocks and everything runs concurrently. This is not to be confused with the same type of concurrency as running in parallel on multiple cores. Javascript is single threaded so each program runs on a single core yet every line of code executes without waiting for anything to return. This sounds weird but it's true. If you want to have any type of sequential ordering you can use events, callbacks, or as of late promises.

jimmytuc

Docker - How to cleanup (unused) resources

Once in a while, you may need to cleanup resources (containers, volumes, images, networks) ...

delete volumes

// see: https://github.com/chadoe/docker-cleanup-volumes

$ docker volume rm $(docker volume ls -qf dangling=true)

$ docker volume ls -qf dangling=true | xargs -r docker volume rm