Skip to content

Instantly share code, notes, and snippets.

Avatar

Stefaan Lippens soxofaan

View GitHub Profile
@soxofaan
soxofaan / ensure_pyspark_executor_logging.py
Created Apr 7, 2020
Decorator based trick to enable Python logging from PySpark executors
View ensure_pyspark_executor_logging.py
import functools
import logging
import pyspark
from typing import Callable
LOG_FORMAT = "[P%(process)s/%(name)s] %(levelname)s: %(message)s"
def ensure_executor_logging(
f=None, *,
@soxofaan
soxofaan / openeo-mask-example.ipynb
Last active Mar 27, 2020
openeo-mask-example.ipynb
View openeo-mask-example.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@soxofaan
soxofaan / Issue107-graph-merging.ipynb
Created Feb 10, 2020
openeo python client issue #107 example
View Issue107-graph-merging.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@soxofaan
soxofaan / pyrasite-stacktrace-how-to.md
Last active Aug 31, 2020 — forked from reywood/how-to.md
How to get a stack trace from a stuck/hanging python script
View pyrasite-stacktrace-how-to.md

How to get a stack trace for each thread in a running Python script

Sometimes a Python script will simply hang forever with no indication of what is going wrong. Perhaps it's polling a service that will never return a value that allows the program to move forward.

Here's a way to see where the program is currently stuck, using pyrasite a tool for injecting code into running Python processes.

Install gdb and pyrasite

Install gdb.

View openeo-mask-example.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View luca-d22.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View multiple-result-nodes-bug.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@soxofaan
soxofaan / openeo-vito-binary-download.ipynb
Created Dec 10, 2019
openEO VITO download binary mask
View openeo-vito-binary-download.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View openeo-scale-range-resample-issue.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@soxofaan
soxofaan / here-jupyter.sh
Last active Aug 27, 2019
here-jupyter: launch Jupyter Notebook in background from current directory using tmux
View here-jupyter.sh
#!/bin/bash
# Build tmux session name from current directory
SESSION_NAME=$(pwd)
# Strip non-alphanumeric characters
SESSION_NAME=${SESSION_NAME//[^a-zA-Z0-9]/}
# Only keep trailing part
SESSION_NAME=Jupyter-${SESSION_NAME: -20}
# Command to run inside tmux session
You can’t perform that action at this time.