Skip to content

Instantly share code, notes, and snippets.

View bthaman's full-sized avatar

Bill Thaman bthaman

  • HDR
  • Austin
View GitHub Profile
@bthaman
bthaman / df_to_sql_fast.py
Created January 17, 2022 19:08
High-performance Pandas dataframe to SQL Server - uses pyodbc executemany with fast_executemany = True. This is an alternative to out-of-the-box Pandas df_to_sql, which is slow for larger dataframes.
def df_to_sql_fast(df, table_name, numeric_columns, date_columns, append_or_replace, conn):
"""
Appends or overwrites a SQL Server table
using data from a Pandas DataFrame.
Submits df records at once for faster performance
compared to df_to_sql.
Parameters:
df (DataFrame): df used to create/append table
table_name (str): Name of existing SQL Server table
@bthaman
bthaman / pyodbc_rollback.py
Last active June 15, 2023 01:52
How to do a SQL Server transaction rollback in pyodbc. Demonstrates how to keep your database consistent when performing a series of updates using pyodbc, and something goes wrong somewhere in the middle of it all.
"""
Demonstrate how to keep your SQL Server database consistent when performing a series of updates using pyodbc, and something goes wrong
somewhere in the middle of it all.
Transactions are managed at the connection level (not the cursor level). When creating the connection, set autocommit=False.
When a command (e.g., an update) is executed against the connection, it will not be committed automatically.
If you are executing multiple commands against the connection, and an error is raised before all the commands are complete,
your database may not be consistent. But, since the commands were not committed, it's ok.
@bthaman
bthaman / list_tables_fields.sql
Created December 12, 2018 21:10
For a SQL Server database, lists tables and fields in each for user tables.
SELECT T.name AS Table_Name ,
C.name AS Column_Name ,
P.name AS Data_Type ,
P.max_length AS Size ,
CAST(P.precision AS VARCHAR) + '/' + CAST(P.scale AS VARCHAR) AS Precision_Scale
FROM sys.objects AS T
JOIN sys.columns AS C ON T.object_id = C.object_id
JOIN sys.types AS P ON C.system_type_id = P.system_type_id
WHERE T.type_desc = 'USER_TABLE';
@bthaman
bthaman / gist:95739d3bffe18af8c9620df9f4806fc0
Created November 27, 2018 21:17 — forked from rxaviers/gist:7360908
Complete list of github markdown emoji markup

People

:bowtie: :bowtie: 😄 :smile: 😆 :laughing:
😊 :blush: 😃 :smiley: ☺️ :relaxed:
😏 :smirk: 😍 :heart_eyes: 😘 :kissing_heart:
😚 :kissing_closed_eyes: 😳 :flushed: 😌 :relieved:
😆 :satisfied: 😁 :grin: 😉 :wink:
😜 :stuck_out_tongue_winking_eye: 😝 :stuck_out_tongue_closed_eyes: 😀 :grinning:
😗 :kissing: 😙 :kissing_smiling_eyes: 😛 :stuck_out_tongue:
@bthaman
bthaman / README-Template.md
Created November 19, 2018 18:19 — forked from PurpleBooth/README-Template.md
A template to make good README.md

Project Title

One Paragraph of project description goes here

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

@bthaman
bthaman / useful_pandas_snippets.py
Created November 19, 2018 16:47 — forked from bsweger/useful_pandas_snippets.md
Useful Pandas Snippets
# List unique values in a DataFrame column
# h/t @makmanalp for the updated syntax!
df['Column Name'].unique()
# For each unique value in a DataFrame column, get a frequency count
df['Column Name'].value_counts()
# Convert Series datatype to numeric (will error if column has non-numeric values)
# h/t @makmanalp
pd.to_numeric(df['Column Name'])
@bthaman
bthaman / python_package_paths.py
Last active November 19, 2018 17:02
Command to determine where Python searches for packages. Get all the details from https://leemendelowitz.github.io/blog/how-does-python-find-packages.html
import sys
'''
list paths that Python searches for packages.
see https://leemendelowitz.github.io/blog/how-does-python-find-packages.html for the whole story.
an example of a path Python searches are the paths listed in the PYTHONPATH environment variable.
'''
print('\n'.join(sys.path))
@bthaman
bthaman / download_path.py
Created November 19, 2018 15:49
Python function that returns the download path for the current user in Windows or Linux (e.g., 'C:\Users\username\Downloads')
import os
def get_download_path():
"""Returns the default downloads path for linux or windows"""
if os.name == 'nt':
import winreg
sub_key = r'SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders'
downloads_guid = '{374DE290-123F-4565-9164-39C4925E467B}'
with winreg.OpenKey(winreg.HKEY_CURRENT_USER, sub_key) as key:
@bthaman
bthaman / get_pdf.py
Created November 19, 2018 15:33
Python function to download a pdf from the web
import requests
def download_file(url, out_file_name):
try:
pdf_url = url
r = requests.get(pdf_url)
with open(out_file_name, 'wb') as f:
f.write(r.content)
@bthaman
bthaman / genetic_algo_projects.py
Last active November 26, 2018 20:33
Demonstration of using a genetic algorithm to find a potentially optimum combination of projects where the sum of costs is at or less than a defined total cost.
"""
author: Bill Thaman
description: Uses a genetic algorithm, with selective pressure, to find the optimal combination
of projects where the sum of costs is less than or equal to a total. Inspired by the knapsack problem.
"""
from random import randint, random
from operator import add
from functools import reduce
import roulette
import project