yurukov

yurukov

Requires proj4 and simplify-geojson. Receives coordinates in EPSG:8122 projection as regularly used in Bulgarian arcgeo government servers. Transforms to geojson format with reduced precision of less than a meter. Then if the size of the resulting file is bigger than 20k characters, it simplifies it to reduce load on processing later on.

yurukov

yurukov

wget -O plovdiv.html "http://transport.plovdiv.bg/desktop/"
grep -o '"number"[^}]\+' plovdiv.html | sed 's_"number":__;s_"\?,"\(name\|lat\|lng\)":"\?_\t_g;s_'"'"'_"_g;s_\\\\__g' | sort -n > stops.tsv
grep 'zul.sel.Listcell' plovdiv.html | grep -v "label:'Линия" | sed "s_\['zul[^{]*{__" | tr '\n' '|' | sed 's_,\[\]\],|__g' | tr '|' '\n' | sed "s_'}label.*'}label:'_\t_;s_'}label:.*_\t_;s_label:'[0-9]*\|'},\[\]\]\]\].*\| __g" > lines.tsv
echo -e "id\tname\tlat\tlng\tlines" > plovdiv.tsv
paste stops.tsv lines.tsv | sed 's_\t\t_\t_;s_"_""_g' | awk -F "\t" '{print $1"\t\""$2"\"\t"$3"\t"$4"\t"$5}'  >> plovdiv.tsv

yurukov

These are a few commands that could be used to scrape a full group page
from Facebook. One can use the Graph API, but there some users would be
hidden. The JS commands should be run in a browser and scroll through
the page opening up hidden content and comments. I used Chrome. Once
enough content is opened, you should save the page as any other and
analyse it's contents. 

yurukov

24102
39493
83017
73780
35746
65543
07257
35064
05009
72693

yurukov

wget https://www.sofiatraffic.bg/interactivecard/lines/1 -O a1
wget https://www.sofiatraffic.bg/interactivecard/lines/2 -O a2
wget https://www.sofiatraffic.bg/interactivecard/lines/3 -O a3
grep -h -o 'for="line[0-9]\+">[^<]\+' a* | sed 's_.*line__;s_">_\t_' > lines
mkdir l
for i in `sed 's_\t.*__' lines|sort -n`; do wget -q -O "l/$i.json" "https://www.sofiatraffic.bg/interactivecard/lines/stops/geo?line_id=$i"; echo -n "."; done
for i in l/*; do echo -e `cat $i` | sed 's_{"geometry":_\n{"geometry":_g' > temp; mv temp $i; done
echo '{"type": "FeatureCollection", "features": [' > stops.geojson
grep -h geometry l/* | sed 's_}}]}_}},_' >> stops.geojson
sed -i '$ s_}},_}}]}_' stops.geojson

yurukov

wget -O stops.json "https://varnatraffic.com/Ajax/GetStations" 
echo -n '{"type":"FeatureCollection","features":' > stops.geojson
sed 's_{"id"_\n{"type":"Feature","properties":{"id"_g;s_"text"_"name"_g;s_,"position":{"lat":_},"geometry":{"type":"Point","coordinates":[_g;s_,"lon":_,_g;s_}}_]}}_g' stops.json | sed 's_\[\([0-9.]\+\),\([0-9.]\+\)\]_[\2,\1]_g' >> stops.geojson
echo '}' >> stops.geojson
wget -O site https://varnatraffic.com
grep -o '/bg/Line/Routes/[^"]\+' site  | sed 's_.*/__'| sort -un > linenumbers
for i in `cat linenumbers`; do echo -en $i"\t"; wget -q -O - "https://varnatraffic.com/Ajax/GetLineState?line=$i&direction=0" | grep -o '"id":[0-9]\+,"text"'  | sed 's_"id":\|,"text"__g' | tr '\n' ',' | sed 's_,$__'; echo ""; done > lines.tsv

yurukov

This is a small util for marking neighboring municipalities.

Dependencies:
jquery, leaflet v101

Data:
https://github.com/yurukov/Bulgaria-geocoding/blob/master/municipalities.geojson

When a municipality is marked as red, you need to click on all blue municialities
which are its neighbors. when that's done, you click on the red municipality itself

yurukov

function download(filename, text) {
  var element = document.createElement('a');
  element.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text));
  element.setAttribute('download', filename);
  element.style.display = 'none';
  document.body.appendChild(element);
  element.click();
  document.body.removeChild(element);
}