kuanb
/ groupby_after_apply.py
Last active
July 5, 2017 16:30
Dealing with (attempting to circumvent) workers getting caught up in long Shapely operations
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| from dask.distributed import Client | |
| from dask.dataframe.utils import make_meta | |
| import dask.dataframe as dd | |
| import math | |
| import numpy as np | |
| import pandas as pd | |
| from shapely.wkt import loads | |
| # change for different ref location for geometries csv |
kuanb
/ consolidated_worker_sched_logs_2.txt
Last active
July 5, 2017 23:05
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Note: Looks like workers completely crashed, were restarted (by the Docker container within which they were running) and then got out of sync (?). | |
| ... | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Register tcp://10.0.0.132:38621 | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Register tcp://10.0.0.132:39862 | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:42596 | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:38847 | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:38621 | |
| Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:39862 | |
| Jul 05 15:14:10 ip-10-0-0-216 dhclient: DHCPREQUEST of 10.0.0.216 on ens3 to 10.0.0.1 port 67 (xid=0x6212458 |
kuanb
/ timing_geoseries_distance_calc.py
Created
July 6, 2017 00:29
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from copy import copy | |
| import geopandas as gpd | |
| from shapely.wkt import loads | |
| geodataset_geom_count = 30000 | |
| # two generic shapes | |
| a = loads('MULTIPOLYGON (((649141.3988551921 119588.0508027334, 649205.3394444393 119586.6252604292, 649204.4036520078 119547.1810296645, 649203.3471333382 119400.7549073702, 649137.6164607083 119401.7231623645, 649141.3988551921 119588.0508027334)))') | |
| b = loads('MULTIPOLYGON (((638329.3738409473 104950.4713290278, 637929.373791253 104951.4454448997, 637929.3737395091 105287.3536795844, 638329.3738592704 105287.3537065834, 638329.3738409473 104950.4713290278)))') |
kuanb
/ accessibility_analysis_example.ipynb
Created
July 12, 2017 18:20
Companion notebook for http://kuanbutts.com/2017/07/12/easy-accessibility/
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kuanb
/ spatial_index_and_bounds_vectorization_benchmarks.ipynb
Created
July 13, 2017 02:17
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kuanb
/ determine_straightness.ipynb
Created
July 25, 2017 21:13
Get the "straightness" of a given path
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kuanb
/ mean_median_headways.ipynb
Last active
August 6, 2017 19:51
Companion to https://twitter.com/buttsmeister/status/893027567896150017
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kuanb
/ index.html
Last active
August 10, 2017 20:15
Bl.ock to play with line offsetting in Mapbox GL.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html> | |
| <head> | |
| <meta charset='utf-8' /> | |
| <title></title> | |
| <meta name='viewport' content='initial-scale=1,maximum-scale=1,user-scalable=no' /> | |
| <script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.38.0/mapbox-gl.js'></script> | |
| <link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.38.0/mapbox-gl.css' rel='stylesheet' /> | |
| <style> | |
| body { margin:0; padding:0; } |
kuanb
/ cleaning_flocktracker_bogota.ipynb
Created
August 12, 2017 19:02
Companion notebook to notes on cleaning Flocktracker data
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kuanb
/ triangulate-spline.ipynb
Created
August 19, 2017 00:12
Companion notebook to this blog post: http://kuanbutts.com/2017/08/18/triangulate-spline/
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.