| #!/bin/bash | |
| # Create a folder called data | |
| mkdir -p data | |
| # Find the most recent parquet directory with the theme 'places' | |
| recent_dir=$(aws s3 ls s3://overturemaps-us-west-2/release/ --recursive | grep "theme=places" | sort | tail -n 1 | awk -F '/' '{print $1"/"$2"/"$3"/"$4}') | |
| # Extract the release date from the directory path | |
| release_date=$(echo $recent_dir | awk -F '/' '{print $2}') |
General:
Cooking:
Correctly prioritizing and targeting performance problems and optimization opportunities is one of the hardest things to master in programming. There are a lot of ways to do it wrong: by prematurely optimizing non-bottlenecks, or preferring fast solutions to clear solutions, or measuring problems incorrectly.
I'll try to summarize what I've learned about doing this right.
First, don't optimize until there's an issue. And issues should be defined as application issues: performance problems that are either detectable by the users (lag) or endanger the platform – i.e. problems that cause downtime, like out-of-memory issues. Until there's an issue, don't think about peformance at all: just solve the problem at hand, which is "creating value for the end-user," or some less-corporate translation of the same.
Second, only optimize with instruments. By instruments, I mean technology that lets you decipher which sub-part of the stack is the bottleneck. Let's say you see slowness around fet
| # TODO: replace :token, :user, and :repo | |
| curl -H "Authorization: token :token" \ | |
| -H 'Accept: application/vnd.github.everest-preview+json' \ | |
| "https://api.github.com/repos/:user/:repo/dispatches" \ | |
| -d '{"event_type": "awesomeness", "client_payload": {"foo": "bar"}}' |
I've been deceiving you all. I had you believe that Svelte was a UI framework — unlike React and Vue etc, because it shifts work out of the client and into the compiler, but a framework nonetheless.
But that's not exactly accurate. In my defense, I didn't realise it myself until very recently. But with Svelte 3 around the corner, it's time to come clean about what Svelte really is.
Svelte is a language.
Specifically, Svelte is an attempt to answer a question that many people have asked, and a few have answered: what would it look like if we had a language for describing reactive user interfaces?
A few projects that have answered this question:
| FROM python:3.6-slim-stretch as csvbuilder | |
| # This one uses csvs-to-sqlite to compile the DB, and then uses datasette | |
| # inspect to generate inspect-data.json Compiling pandas takes way too long | |
| # under alpine so we use slim-stretch for this one instead. | |
| RUN apt-get update && apt-get install -y python3-dev gcc | |
| COPY *.csv csvs/ | |
| RUN pip install csvs-to-sqlite datasette | |
| RUN csvs-to-sqlite csvs/names.csv data.db -f "name" -c "legislature" -c "country" |
Two simple examples of quickly loading data into PostgreSQL with the COPY command.
Much more sophisticated stuff is possible. Refer to the official docs and consider adopting a wrapper like django-postgres-copy for more complicated tasks.
| pg_dump -Fc --no-acl --no-owner -h localhost -U <YOUR LOCAL DATABASE USER> <YOUR LOCAL DATABASE NAME> > <YOUR LOCAL DATABASE NAME>.dump; | |
| pg_restore -v -h <YOUR RDS IP ADDRESS> -U <YOUR RDS DATABASE USER> -d <YOUR RDS DATABASE NAME> <YOUR LOCAL DATABASE NAME>.dump; |
The PATH is an important concept when working on the command line. It's a list
of directories that tell your operating system where to look for programs, so
that you can just write script instead of /home/me/bin/script or
C:\Users\Me\bin\script. But different operating systems have different ways to
add a new directory to it:
