Requires proj4 and simplify-geojson. Receives coordinates in EPSG:8122 projection as regularly used in Bulgarian arcgeo government servers. Transforms to geojson format with reduced precision of less than a meter. Then if the size of the resulting file is bigger than 20k characters, it simplifies it to reduce load on processing later on.
View bash commands
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wget -O stops.json "https://varnatraffic.com/Ajax/GetStations" | |
echo -n '{"type":"FeatureCollection","features":' > stops.geojson | |
sed 's_{"id"_\n{"type":"Feature","properties":{"id"_g;s_"text"_"name"_g;s_,"position":{"lat":_},"geometry":{"type":"Point","coordinates":[_g;s_,"lon":_,_g;s_}}_]}}_g' stops.json | sed 's_\[\([0-9.]\+\),\([0-9.]\+\)\]_[\2,\1]_g' >> stops.geojson | |
echo '}' >> stops.geojson | |
wget -O site https://varnatraffic.com | |
grep -o '/bg/Line/Routes/[^"]\+' site | sed 's_.*/__'| sort -un > linenumbers | |
for i in `cat linenumbers`; do echo -en $i"\t"; wget -q -O - "https://varnatraffic.com/Ajax/GetLineState?line=$i&direction=0" | grep -o '"id":[0-9]\+,"text"' | sed 's_"id":\|,"text"__g' | tr '\n' ',' | sed 's_,$__'; echo ""; done > lines.tsv |
View bash commands
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wget -O plovdiv.html "http://transport.plovdiv.bg/desktop/" | |
grep -o '"number"[^}]\+' plovdiv.html | sed 's_"number":__;s_"\?,"\(name\|lat\|lng\)":"\?_\t_g;s_'"'"'_"_g;s_\\\\__g' | sort -n > stops.tsv | |
grep 'zul.sel.Listcell' plovdiv.html | grep -v "label:'Линия" | sed "s_\['zul[^{]*{__" | tr '\n' '|' | sed 's_,\[\]\],|__g' | tr '|' '\n' | sed "s_'}label.*'}label:'_\t_;s_'}label:.*_\t_;s_label:'[0-9]*\|'},\[\]\]\]\].*\| __g" > lines.tsv | |
echo -e "id\tname\tlat\tlng\tlines" > plovdiv.tsv | |
paste stops.tsv lines.tsv | sed 's_\t\t_\t_;s_"_""_g' | awk -F "\t" '{print $1"\t\""$2"\"\t"$3"\t"$4"\t"$5}' >> plovdiv.tsv |
View bash commands
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wget https://www.sofiatraffic.bg/interactivecard/lines/1 -O a1 | |
wget https://www.sofiatraffic.bg/interactivecard/lines/2 -O a2 | |
wget https://www.sofiatraffic.bg/interactivecard/lines/3 -O a3 | |
grep -h -o 'for="line[0-9]\+">[^<]\+' a* | sed 's_.*line__;s_">_\t_' > lines | |
mkdir l | |
for i in `sed 's_\t.*__' lines|sort -n`; do wget -q -O "l/$i.json" "https://www.sofiatraffic.bg/interactivecard/lines/stops/geo?line_id=$i"; echo -n "."; done | |
for i in l/*; do echo -e `cat $i` | sed 's_{"geometry":_\n{"geometry":_g' > temp; mv temp $i; done | |
echo '{"type": "FeatureCollection", "features": [' > stops.geojson | |
grep -h geometry l/* | sed 's_}}]}_}},_' >> stops.geojson | |
sed -i '$ s_}},_}}]}_' stops.geojson |
View Marking neighboring municipalities
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a small util for marking neighboring municipalities. | |
Dependencies: | |
jquery, leaflet v101 | |
Data: | |
https://github.com/yurukov/Bulgaria-geocoding/blob/master/municipalities.geojson | |
When a municipality is marked as red, you need to click on all blue municialities | |
which are its neighbors. when that's done, you click on the red municipality itself |
View scraper.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function download(filename, text) { | |
var element = document.createElement('a'); | |
element.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text)); | |
element.setAttribute('download', filename); | |
element.style.display = 'none'; | |
document.body.appendChild(element); | |
element.click(); | |
document.body.removeChild(element); | |
} |
View map.geojson
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View query.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SELECT a.timest,a.level, a.isinvalid, (select round(avg(a1.level)) from air_sofia a1 where a1.station=a.station and a1.param=0 and a1.isinvalid=0 and a1.timest>=a.timest - interval 12 hour and a1.timest<a.timest + interval 12 hour) level24,(select count(a2.level) from air_sofia a2 where a2.station=a.station and a2.param=0 and a2.isinvalid=0 and a2.timest>=a.timest - interval 12 hour and a2.timest<a.timest + interval 12 hour) validcount FROM air_sofia a where param=0 and station=1 |
View README
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Open http://registry.plovdiv.bg/eDogs/default.aspx | |
2. Select 50 entries per page | |
3. Copy JS in console and run | |
4. Back up the downloaded data file | |
5. Extract all addresses: | |
awk -F '\t' '{print $4}' plddogs.tsv | grep -v address | sed 's_,\? \?\(ет\|ап\)\.\? \?[0-9]\+ \?__g;s_ \?№ \?_ _' |sort -u > addr | |
6. Geotag the addresses: | |
php geotag.php addr > addrg | |
7. Merge geotagged addresses with scaraped data and exclude columns: | |
for i in `sed 's_\t_|_g;s_ _\__g' plddogs.tsv | grep -v address`; |
View README
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I updated my permalink structure from .../Y/m/d/title to .../Y/title. | |
Naturaly the share button count was set to 0 due to the new URL. My | |
solution was to use the old URL structure for the share buttons only | |
for posts before the update. The downside of this is that old posts | |
will be shared with the old url structure. Thus when shared, there'll | |
be another 301. Upside - share count is kept. | |
Also, I wanted to optimize the scripts written to the footer by the | |
share module of jetpack. The registered js file includes all the | |
scripts for opening the share dialogs in a new window. These are the |
NewerOlder