This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.
View .dockerfile
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM python:3.8.5-slim-buster as base | |
LABEL maintainer="Your Name <Your@Name.com>" \ | |
version="1.0.0" | |
RUN apt-get update && apt-get install tzdata -y --no-install-recommends | |
ENV TZ="America/Los_Angeles" | |
ENV PIP_NO_CACHE_DIR=1 | |
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 |
View ads1115_differential.ino
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// ads1115_differential.ino | |
// PURPOSE: read differential | |
// EDITED BY: JFS 03/26/21 | |
// AUTHOR: Rob.Tillaart | |
// VERSION: 0.1.1 | |
// test 1 | |
// +signal on AIN0, -signal on AIN1 | |
//EXAMPLE OUTPUT |
View stream_arduino.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
""" | |
stream_arduino.py | |
Stream data from an arduino on a com port | |
must run %matplotlib | |
data comes in this format: | |
12351\r\n | |
Written by: John Franco Saraceno |
View git_update.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
if [ $# != 2 ]; then | |
echo "\nchris k's git update automater sh script. Edit to put a bunch of \`git add blah'" | |
echo "lines in the body then evoke with the branch name and commit description\n" | |
echo "Usage: $0 <branch_name> <\"Description of update\">\n" | |
echo " branch_name: the name of the git branch to be created" | |
echo " Description: Text for: git commit -m \"Description of update\"" | |
echo "" | |
echo "Example:" | |
BRANCH="core_override" |
View xmas_ornament.ino
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <Adafruit_NeoPixel.h> | |
#define N_PIXELS 19 | |
#define LED_PIN 1 | |
int RandomBlinky = 0; | |
int Brightness = 120; // Brightness of the pixels is about 60% | |
int BlinkDelay = 15; | |
int i = 0; |
View read_exo_korv2.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
def read_exo_korv2(fname): | |
df = pd.read_csv(fname, skiprows=9, encoding="UTF-16", parse_dates=[[0,1]], | |
index_col=0, keep_date_col=False) | |
return df | |
datafile = r'./KOR_measurement_file_some_date.csv' | |
df = read_exo_korv2(datafile) |
View exif_gps.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import exifread | |
# based on https://gist.github.com/erans/983821 | |
def _get_if_exist(data, key): | |
if key in data: | |
return data[key] | |
return None |
View unzip_r_packages_cli.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
""" | |
Created on Wed Jun 6 11:51:58 2018 | |
unzip a directory of zip files to a destination directory using the a cli | |
USAGE: | |
$ python unzip_r_packages_cli.py --indir path_to_zip_files --outdir path_to_destination | |
@author: jsaracen | |
""" |
View unpack_zips.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
""" | |
Created on Wed Jun 6 11:51:58 2018 | |
unzip a directory of zip files to a destination directory | |
@author: jsaracen | |
""" | |
import zipfile | |
import os | |
NewerOlder