-
jq — https://jqlang.github.io/jq/ — "like sed for JSON data"
There are several options available for installing jq. I prefer to use Homebrew:
brew install jq
class Singleton(object): | |
def __new__(cls): | |
if not hasattr(cls, 'instance') or not cls.instance: | |
cls.instance = super().__new__(cls) | |
return cls.instance | |
obj1 = Singleton() | |
obj2 = Singleton() |
I screwed up using git ("git checkout --" on the wrong file) and managed to delete the code I had just written... but it was still running in a process in a docker container. Here's how I got it back, using https://pypi.python.org/pypi/pyrasite/ and https://pypi.python.org/pypi/uncompyle6
apt-get update && apt-get install gdb
This was tested on a ThinkPad P70 laptop with an Intel integrated graphics and an NVIDIA GPU:
lspci | egrep 'VGA|3D'
00:02.0 VGA compatible controller: Intel Corporation Device 191b (rev 06)
01:00.0 VGA compatible controller: NVIDIA Corporation GM204GLM [Quadro M3000M] (rev a1)
A reason to use the integrated graphics for display is if installing the NVIDIA drivers causes the display to stop working properly.
In my case, Ubuntu would get stuck in a login loop after installing the NVIDIA drivers.
This happened regardless if I installed the drivers from the "Additional Drivers" tab in "System Settings" or the ppa:graphics-drivers/ppa
in the command-line.
____________________________________________________________________________________________________ | |
Layer (type) Output Shape Param # Connected to | |
==================================================================================================== | |
input_1 (InputLayer) (None, 32, 10) 0 | |
____________________________________________________________________________________________________ | |
lstm_1 (LSTM) (None, 32, 10) 840 input_1[0][0] | |
____________________________________________________________________________________________________ | |
add_1 (Add) (None, 32, 10) 0 input_1[0][0] | |
lstm_1[0][0] | |
____________________________________________________________________________________________________ |
// Based on following links: | |
// http://andrew.hedges.name/experiments/haversine/ | |
// http://www.movable-type.co.uk/scripts/latlong.html | |
df | |
.withColumn("a", pow(sin(toRadians($"destination_latitude" - $"origin_latitude") / 2), 2) + cos(toRadians($"origin_latitude")) * cos(toRadians($"destination_latitude")) * pow(sin(toRadians($"destination_longitude" - $"origin_longitude") / 2), 2)) | |
.withColumn("distance", atan2(sqrt($"a"), sqrt(-$"a" + 1)) * 2 * 6371) | |
>>> | |
+--------------+-------------------+-------------+----------------+---------------+----------------+--------------------+---------------------+--------------------+------------------+ | |
|origin_airport|destination_airport| origin_city|destination_city|origin_latitude|origin_longitude|destination_latitude|destination_longitude| a| distance| |
Git for Windows comes bundled with the "Git Bash" terminal which is incredibly handy for unix-like commands on a windows machine. It is missing a few standard linux utilities, but it is easy to add ones that have a windows binary available.
The basic idea is that C:\Program Files\Git\mingw64\
is your /
directory according to Git Bash (note: depending on how you installed it, the directory might be different. from the start menu, right click on the Git Bash icon and open file location. It might be something like C:\Users\name\AppData\Local\Programs\Git
, the mingw64
in this directory is your root. Find it by using pwd -W
).
If you go to that directory, you will find the typical linux root folder structure (bin
, etc
, lib
and so on).
If you are missing a utility, such as wget, track down a binary for windows and copy the files to the corresponding directories. Sometimes the windows binary have funny prefixes, so
import findspark | |
findspark.init("[spark install location]") | |
import pyspark | |
import string | |
from pyspark import SparkContext | |
from pyspark.sql import SQLContext | |
from pyspark.mllib.util import MLUtils | |
from pyspark.sql.types import * | |
from pyspark.ml.feature import CountVectorizer, CountVectorizerModel, Tokenizer, RegexTokenizer, StopWordsRemover |
#!/bin/bash | |
set +x | |
# Setup all of the pyenvs | |
export PATH="$HOME/.pyenv/bin:$PATH" | |
export CONFIGURE_OPTS='--enable-shared' | |
eval "$(pyenv init -)" | |
pyenv shell 2.7.5 | |
pyenv virtualenvwrapper | |
pyenv rehash |