Skip to content

Instantly share code, notes, and snippets.

View sebastianschramm's full-sized avatar

Sebastian M. Schramm sebastianschramm

10 followers · 21 following
View GitHub Profile
@sebastianschramm
sebastianschramm / pathlib_mkdir.py
Last active August 12, 2022 18:14
How to create a directory and all of it's parent directories if none or some of them do not exist yet?
from pathlib import Path
data_directory_path = Path('data/assets')
data_directory_path.mkdir(parents=True, exist_ok=True)
@sebastianschramm
sebastianschramm / boto3_s3_upload_tqdm_progressbar.py
Created August 13, 2022 09:17
Show tqdm progress bar while uploading file to S3 with boto3
from pathlib import Path
import boto3
from tqdm import tqdm
def upload_to_s3(
file_name: str,
target_bucket_name: str,
target_object_name: str,
@sebastianschramm
sebastianschramm / pydantic_model_parse_obj.py
Created August 13, 2022 18:39
Instantiate a pydantic BaseModel from a dictionary object
from datetime import datetime
from pydantic import BaseModel
class Message(BaseModel):
id: int
content: str
created_at: datetime
@sebastianschramm
sebastianschramm / combine_2_dicts.py
Created August 14, 2022 12:49
Two ways to combine two dictionaries with unique keys
first_dict = {"a": 1, "b": "foo"}
second_dict = {"c": 1, "d": 5.5, "e": "bar"}
combined_dict_1 = first_dict | second_dict
combined_dict_2 = {**first_dict, **second_dict}
assert combined_dict_1 == combined_dict_2
@sebastianschramm
sebastianschramm / path_delete_all_files_in_directory.py
Created August 16, 2022 12:56
How to get all files in directory and delete them?
from pathlib import Path
def delete_all(directory_path: Path):
if directory_path.is_dir():
for file in directory_path.iterdir():
file.unlink(missing_ok=True)
if __name__ == "__main__":
@sebastianschramm
sebastianschramm / groupby_get_unique_dicts.py
Created August 17, 2022 15:44
How to get set of dictionaries from a list of duplicate dictionaries?
import operator
from itertools import groupby
list_of_city_dicts = [
{"city": "New York"},
{"city": "Hamburg"},
{"city": "New York"},
{"city": "Puerto Viejo"},
]
@sebastianschramm
sebastianschramm / groupby_counter_dictionaries.py
Created August 18, 2022 15:24
How to get a Counter of dictionaries from a list of dictionaries?
import operator
from itertools import groupby
list_of_city_dicts = [
{"city": "New York", "country": "US"},
{"city": "San José", "country": "CR"},
{"city": "New York", "country": "US"},
{"city": "San José", "country": "CO"},
{"city": "Puerto Viejo", "country": "CR"},
]
@sebastianschramm
sebastianschramm / moto_s3_integration_test.py
Created August 19, 2022 14:56
An easy way to integration test your calls to S3 with moto
import boto3
from moto import mock_s3
def create_s3_bucket(bucket_name: str, aws_region: str):
s3_client = boto3.client("s3", region_name=aws_region)
s3_client.create_bucket(
Bucket=bucket_name, CreateBucketConfiguration={"LocationConstraint": aws_region}
)
@sebastianschramm
sebastianschramm / cloudpathlib_handling_s3_paths.py
Created August 20, 2022 10:12
How to handle S3 paths using cloudpathlib
from pathlib import Path
from cloudpathlib import CloudPath
s3_file = "s3://mybucket/foo/bar.txt"
cloud_path = CloudPath(s3_file)
conventional_path = Path(s3_file)
"""
@sebastianschramm
sebastianschramm / dict_fromkeys.py
Last active August 21, 2022 11:32
How to initialize a dictionary from a list of keys?
feature_names = ["sepal_length", "sepal_width", "petal_length", "petal_width"]
default_type = float
feature_map = dict.fromkeys(feature_names, default_type)
"""
>>> feature_map
{'sepal_length': <class 'float'>, 'sepal_width': <class 'float'>,
'petal_length': <class 'float'>, 'petal_width': <class 'float'>}