Skip to content

Instantly share code, notes, and snippets.

View drorata's full-sized avatar

Dror Atariah drorata

View GitHub Profile
@drorata
drorata / gist:146ce50807d16fd4a6aa
Last active February 27, 2024 10:15
Minimal Working example of Elasticsearch scrolling using Python client
# Initialize the scroll
page = es.search(
index = 'yourIndex',
doc_type = 'yourType',
scroll = '2m',
search_type = 'scan',
size = 1000,
body = {
# Your query's body
})
@drorata
drorata / send_email.py
Created September 27, 2017 13:45
Minimal example of sending a JSON over email
import json
# Import smtplib for the actual sending function
import smtplib
# Import the email modules we'll need
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from datetime import datetime
def send_email(data):
@drorata
drorata / notebook.ipynb
Created May 10, 2023 09:42
Spark comparing columns
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@drorata
drorata / python-build.20230404093016.30887.log
Last active April 4, 2023 06:35
Fail to build 3.8.9 using Pyenv
/var/folders/y6/skpcb0d11fb0yknzv6934h480000gn/T/python-build.20230404093016.30887 ~/tmp
++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:334): http(): local method=get
++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:335): http(): '[' -n https://www.python.org/ftp/python/3.8.9/Python-3.8.9.tar.xz ']'
++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:336): http(): shift 1
+++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:338): http(): detect_http_client
+++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:345): detect_http_clienlocal client
+++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:346): detect_http_clienfor client in aria2c curl wget
+++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:347): detect_http_clientype aria2c
+++(/opt/homebrew/Cellar/pyenv/2.3.16/plugins/python-build/bin/python-build:346): detect_http_clienfor client
Visual Studio Code (1.74.3, undefined, desktop)
Jupyter Extension Version: 2023.1.1003441034.
Python Extension Version: 2022.20.2.
Workspace folder /Users/username/My Drive/Private/Football analysis
warn 10:36:00.299: Exception while attempting zmq : No native build was found for platform=darwin arch=arm64 runtime=electron abi=106 uv=1 armv=8 libc=glibc node=16.14.2 electron=19.1.8
loaded from: /Users/username/.vscode/extensions/ms-toolsai.jupyter-2023.1.1003441034/out/node_modules/zeromq
User belongs to experiment group 'jupyterTest'
User belongs to experiment group 'jupyterEnhancedDataViewer'
debug 10:36:00.454: Updating cache with Local kernels
@drorata
drorata / gist:608739e1aa4e5f31a62cbd63455c99fd
Created January 9, 2023 06:31
Detailed logs of Jupyter inside VScode
Visual Studio Code (1.74.2, undefined, desktop)
Jupyter Extension Version: 2023.1.1003441034.
Python Extension Version: 2022.20.1.
Workspace folder /Users/username/My Drive/Private/Football analysis
warn 07:29:47.904: Exception while attempting zmq : No native build was found for platform=darwin arch=arm64 runtime=electron abi=106 uv=1 armv=8 libc=glibc node=16.14.2 electron=19.1.8
loaded from: /Users/username/.vscode/extensions/ms-toolsai.jupyter-2023.1.1003441034/out/node_modules/zeromq
User belongs to experiment group 'jupyterTest'
User belongs to experiment group 'jupyterEnhancedDataViewer'
debug 07:29:48.056: Updating cache with Local kernels
Experiment 'pythonaacf' is active
Experiment 'pythonSurveyNotificationcf' is active
> poetry env list --full-path
cwd: ~/My Drive/Private/Football analysis
[DEBUG 2023-0-6 11:45:15.870]: Cached data exists KeyPrefix=Cache_Method_Output_m.addKeyToStorage-Args="global"-Arg-Separator-"WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/username/My Drive/Private/Football analysis"-Arg-Separator-undefined
[DEBUG 2023-0-6 11:45:15.870]: Cached data exists KeyPrefix=Cache_Method_Output_m.addKeyToStorage-Args="workspace"-Arg-Separator-"autoSelectedWorkspacePythonInterpreter-/Users/username/My Drive/Private/Football analysis"-Arg-Separator-undefined
[DEBUG 2023-0-6 11:45:15.870]: Cached data exists KeyPrefix=Cache_Method_Output_m.addKeyToStorage-Args="workspace"-Arg-Separator-"autoSelectedWorkspacePythonInterpreter-/Users/username/My Drive/Private/Football analysis"-Arg-Separator-undefined
[DEBUG 2023-0-6 11:45:15.871]: Cached data exists KeyPrefix=Cache_Method_Output_m.addKeyToStorage-Args="global"-Arg-Separator-"WORKSPACE_FOLD
@drorata
drorata / description.md
Created December 27, 2022 17:24
MWE - rendering Markdown using a LaTeX template

I am using the function render_latex:

def render_latex(text: str, minimal: bool = True) -> str:
    logger.debug(f"Rendering LaTeX - {text}")
    with LaTeXRenderer() as rendered:
        ast = Document(text)
        if len(ast.children) != 1:
            raise ValueError("Number of children of provided text is not 1")
 for child in ast.children[0].children:
@drorata
drorata / example.ipynb
Created November 11, 2022 07:27
PySpark and time zones
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.