Skip to content

Instantly share code, notes, and snippets.

@skseth skseth/osx_setup.md
Last active Jul 21, 2019

Embed
What would you like to do?
os x setup

Setting up my OS X machine

First Things First

  • Enable FileVault (for encryption) - save recovery key in safe place
  • Sophos Anti-virus - update files
  • Office for Mac (if needed for project)

Setting Up a Profile

Based on this stackoverflow answer .bash_profile should be always

if [ -f ~/.profile ]; then
    source ~/.profile
fi

if [ -f ~/.bashrc ]; then
    source ~/.bashrc
fi

.profile should contain stuff you want in every shell
.bashrc should contain only bash-specific stuff for interactive use - command prompts, bash completion etc.

xcode / xcode command line tools

Needed for several of the following tools such as brew, python

xcode-select --install

You can also install xcode directly via safari from : https://developer.apple.com/download/more/. If you use Safari, you can resume downloads.

Repository Managers

  • Install brew as per homebrew instructions

  • brew tap homebrew/cask-versions - to be able to install versions of casks

  • in .profile - export PATH=/usr/local/bin:/usr/local/sbin:$PATH - export HOMEBREW_GITHUB_API_TOKEN=

Languages and Editors

Visual Studio Code

brew cask install visual-studio-code

Intellij IDEA

Install intellij idea CE / Ultimate as needed

git (default on mac)

  • ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
  • eval "$(ssh-agent -s)"
  • Add following file (~/.ssh/config)
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa
  • ssh-add -K ~/.ssh/id_rsa

  • git config --global user.name ""

  • git config --global user.email ""

  • install GitLens extension for vscode

Java

  • JDK setup
brew cask install adoptopenjdk
brew tap AdoptOpenJDK/openjdk (if you want old versions)
brew cask install <version> # e.g. adoptopenjdk11, adoptopenjdk8

Use /usr/libexec/java_home -V to see list of JDKs

Add following to .profile :

export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
  • install Java extension pack for vscode

  • brew install gradle

Go

  • brew install go
  • in .profile
export GOPATH=~/go
export PATH=$GOPATH/bin:$PATH
  • vscode : install go extension, and go tools needed by extension
  • delve : go get -u github.com/go-delve/delve/cmd/dlv

Rust

  • brew install rustup

  • rustup-init

  • in .profile

export PATH="$HOME/.cargo/bin:$PATH"
  • rustup component add rls rust-analysis rust-src # language server and related components
  • install rls extension in vscode

Python3

  • brew install python3

  • install xcode, sudo xcodebuild -license, sudo xcode-select --install

  • pip3 install virtualenv virtualenvwrapper

  • python3 -m pip install --user --upgrade setuptools wheel twine

Configurinng python3 to use specific SSL certificates is surprisingly complicated.

Create a file ~/.ca/ca-bundle.crt : e.g.

Download https://curl.haxx.se/ca/cacert.pem, then append ~/.ca/ca.local.crt

then, in .profile

export SSL_CERT_FILE=$HOME/.ca/ca-bundle.crt
export REQUESTS_CA_BUNDLE=$HOME/.ca/ca-bundle.crt
  • in .profile
# virtualenv and virtualenvwrapper
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
  • mkvirtualenv py3cv4 -p python3 # p3cv4 is env name - you can replace it with your choice

  • pip install pylint (for vscode)

  • install python extension for vscode

Setup python packaging - see https://packaging.python.org/overview/, and checkout this tutorial : https://packaging.python.org/tutorials/packaging-projects/

Python3 - Nachine learning / computer vision

From macOS for deep learning with Python, TensorFlow, and Keras, with updates from installing open cv4

opencv

OpenCV dependencies :

  • brew install cmake pkg-config
  • brew install jpeg libpng libtiff openexr
  • brew install eigen tbb
  • brew install wget

Download and build opencv

cd ~
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.0.0.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.0.0.zip
unzip opencv.zip
unzip opencv_contrib.zip
mv opencv-4.0.0 opencv
mv opencv_contrib-4.0.0 opencv_contrib
cd ~/opencv
mkdir build
cd build
workon cv
cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D PYTHON3_LIBRARY=`python -c 'import subprocess ; import sys ; s = subprocess.check_output("python-config --configdir", shell=True).decode("utf-8").strip() ; (M, m) = sys.version_info[:2] ; print("{}/libpython{}.{}.dylib".format(s, M, m))'` \
    -D PYTHON3_INCLUDE_DIR=`python -c 'import distutils.sysconfig as s; print(s.get_python_inc())'` \
    -D PYTHON3_EXECUTABLE=$VIRTUAL_ENV/bin/python \
    -D BUILD_opencv_python2=OFF \
    -D BUILD_opencv_python3=ON \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D INSTALL_C_EXAMPLES=OFF \
    -D OPENCV_ENABLE_NONFREE=ON \
    -D BUILD_EXAMPLES=ON ..
  • make opencv 4
make -j4 // use make instead of make -j4 if any error occurs
  • install
sudo make install
  • link opencv into virtual environment
cd /usr/local/python/cv2/python-3.7 # this path depends on using python 3.7
sudo mv cv2.cpython-37m-darwin.so cv2.so # the 37m depends on usingh python 3.7
cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.so cv2.so
  • Verify opencv works
cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.so cv2.so

Keras & Tensorflow

pip install scipy pillow
pip install imutils h5py requests progressbar2
pip install scikit-learn scikit-image
pip install matplotlib
touch ~/.matplotlib/matplotlibrc
echo "backend: TkAgg" >> ~/.matplotlib/matplotlibrc
pip install tensorflow
pip install keras

Test keras setup:ß

$ python
>>> import keras
Using TensorFlow backend.
>>>

Node and NPM

brew install node npm install typescript npm install -g @angular/cli npm install -g create-react-app

TBD: vscode setup for angular, react, typescript

flutter / dart / android studio

Flutter on macos

  • Install flutter
  • Add flutter/bin to path
  • install android studio
  • xcode
brew update
brew install --HEAD usbmuxd
brew link usbmuxd
brew install --HEAD libimobiledevice
brew install ideviceinstaller
brew install ios-deploy
brew install cocoapods
pod setup

Install flutter / dart extensions for Intellij, android studio, visual studio code

Scala

brew install scala brew install sbt

vscode : install Scala syntax extension intellij : scala plugin

C/C++

XCode

Ruby

Erlang

TODO

Virtual Machines and Orchestration Engines

Virtualbox

brew cask install virtualbox

Minikube

brew cask install minikube

Ansible

deactivate pip3 install ansible

CA and DNS

DNS

https://medium.com/@kharysharpe/automatic-local-domains-setting-up-dnsmasq-for-macos-high-sierra-using-homebrew-caf767157e43 https://banck.net/2018/12/using-dnsmasq-on-mac-os-for-local-development/

brew install dnsmasq
echo "listen-address=127.0.0.1" >> $(brew --prefix)/etc/dnsmasq.conf
echo "port=5354" >> $(brew --prefix)/etc/dnsmasq.conf
sudo brew services start dnsmasq

sudo mkdir -v /etc/resolver
sudo tee -a /etc/resolver/local >> EOF
port 5354
nameserver 127.0.0.1
EOF

scutil --dns

On the host, you should set the following when creating any vm :

VBoxManage modifyvm "<VM name>" --natdnshostresolver1 on

*.minikube.local

echo "address=/.minikube.local/192.168.64.11" >> $(brew --prefix)/etc/dnsmasq.conf

CA

See https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309

mkdir ~/.ca
cd ~/.ca

# generate ca private key
openssl genrsa -des3 -out rootCA.key 4096

# see the key - optional
openssl rsa -in ca.local.key -noout -text

# extract ca public key - optional
openssl rsa -in ca.local.key -pubout -out ca.local.pubkey 

# generate root CA Cert
openssl req -x509 -new -nodes -key ca.local.key -sha256 -days 1024 -out ca.local.crt

# Per domain (e.g. artifactory.local)
DOMAIN=artifactory.local
openssl genrsa -out $DOMAIN.key 2048

openssl req -new -sha256 \
    -key $DOMAIN.key \
    -subj "/C=IN/ST=KA/O=Myself/CN=$DOMAIN" \
    -reqexts SAN \
    -config <(cat /etc/ssl/openssl.cnf \
        <(printf "\n[SAN]\nsubjectAltName=DNS:$DOMAIN,DNS:*.$DOMAIN")) \
    -out $DOMAIN.csr

# check the request
openssl req -in $DOMAIN.csr -noout -text

# sign the csr

openssl x509 -req \
    -extfile <(printf "subjectAltName=DNS:$DOMAIN,DNS:*.$DOMAIN") \
    -days 365 -in $DOMAIN.csr \
    -sha256 \
    -CA ca.local.crt -CAkey ca.local.key -CAcreateserial \
    -out $DOMAIN.crt

# print contents of certificates
openssl x509 -in $DOMAIN.pem -text

# see certificate being served by a site
echo | openssl s_client -showcerts -connect kibana.host.local:443 | openssl x509 -inform pem -noout -text

nginx

brew install nginx

sudo brew service start nginx

NOTE: sudo is important. Without that nginx cannot be accessed from outside the host vm

Setup server proxies : SSL example at https://gist.github.com/shijij/54c9b21f26c08a15a70c182f03cb15b4

Also see :

https://stackoverflow.com/questions/16042647/whats-the-de-facto-standard-for-a-reverse-proxy-to-tell-the-backend-ssl-is-used

X-Forwarded-Protocol: https X-Forwarded-Ssl: on X-Url-Scheme: https

lsof -nP -i4TCP:443 | grep LISTEN

Unfortunately, on mac, this may not be enough for guest VMs to access nginx on your host machine.

First, the firewall may

Repositories

brew install artifactory

Nexus

Download nexus 3.0 tar.gz file from : https://help.sonatype.com/repomanager3

Note : As of May 2019, brew install nexus installs old version.

Untar the installation into $(brew --prefix)/opt/, and rename to nexus.

To setup as a service, follow instructions here : https://help.sonatype.com/repomanager3/installation/run-as-a-service

Create following file : /Library/LaunchDaemons/om.sonatype.nexus.plist

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.sonatype.nexus</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/opt/nexus/nexus3xx/bin/nexus</string>
        <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>
sudo chown root:wheel /Library/LaunchDaemons/com.sonatype.nexus.plist
sudo chmod 644 /Library/LaunchDaemons/com.sonatype.nexus.plist
sudo launchctl load /Library/LaunchDaemons/com.sonatype.nexus.plist

To change the port nexus listens on, edit /usr/local/opt/nexus/sonatype-work/nexus3/etc/nexus.properties.

You can also set RUN_AS_USER setting in nexus startup script (as shown in the plist file).

run_as_user='<username>'

Setup nginx with ssl as a frontend for nexus (e.g. https://repos.host.local ).

Docker Repository

Setup 2 docker repos :

docker-private on 8101 docker-hub on 8102, as proxy for dockerhub

setup nginx ssl setup in front.

For minikube, copy the ca crt to docker dir :

cat ~/.ca/ca.local.crt | minikube ssh "sudo mkdir -p /etc/docker/certs.d/docker.host.local && sudo tee /etc/docker/certs.d/docker.host.local/ca.crt"

docker login docker.host.local # provide nexus userid, password

#do the same for dockerhub.host.local

Now try this :

docker pull dockerhub.host.local/busybox
docker tag dockerhub.host.local/busybox:latest docker.host.local/busybox:latest
docker push docker.host.local/busybox:latest

pypi

Follow instructions at https://help.sonatype.com/repomanager3/formats/pypi-repositories

Create pypi-proxy, pypi-hosted and pypi-group

On client machine :

create ~/.pip/pip.conf as follows :

[global]
index = http://localhost:8085/repository/pypi-all/pypi
index-url = http://localhost:8085/repository/pypi-all/simple

Run 'pip config list -v' to verify the file is read correctly.

For publishing with pypi, setup ~/.pypirc

[distutils]
index-servers = pypi
[pypi]
repository: http://localhost:8081/repository/pypi-hosted/
username: <userid>
password: <password>

It may work to leave out username/password and enter it while uploading.

  • maven
  • pypi
  • yum
  • apt
  • cargo

Athens (Go) Cargo (Rust) pub (dart)

General development

CA setup

DNS

Service development

ELK Stack

See https://logz.io/blog/elk-mac/.

brew install elasticsearch logstash kibana

  • Elasticsearch

brew services start elasticsearch # verify http://localhost:9200 Check elastic search health : http://localhost:9200/_cluster/health?pretty

If there is no response, there may be an error.

NOTE:

As of May 1, 2019 on Mac High Sierra, there was the following bug : https://discuss.elastic.co/t/elasticsearch-6-7-0-homebrew-install-macos-10-14-4-fails-to-run-error-cluster-name-elasticsearch-nathan-subdirectory-exists-in-data-paths/174747/3

To fix : rm -rf /usr/local/var/lib/elasticsearch/elasticsearch_/

  • Kibana

brew services start kibana

Fix the kibana config :

sudo vi /usr/local/etc/kibana/kibana.yml

uncomment following lines :

server.port: 5601
elasticsearch.hosts: ["http://localhost:9200"]

Check all is working : http://localhost:5601/status

  • Logstash

You can run in

Add a syslog pipeline :

sudo vim /etc/logstash/conf.d/syslog.conf

input {
  file {
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"] 
    index => "syslog-demo"
  }
  stdout { codec => rubydebug }
}

Prometheus

brew install prometheus

Gitlab

Proxying Gitlab : https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl

Also see : https://knowledge.rootknecht.net/integrate-gitlab-ce-docker-with-external-nginx-reverse-proxy

nginx['listen_port'] = 8081 nginx['listen_https'] = false nginx['proxy_set_headers'] = { "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on" }

Also, set https for registry, mattermost, pages

sudo gitlab-ci

MySQL

redis

cassandra

kafka

GRPC / OpenAPI

Istio

Envoy

Machine Learning development

Airflow / Cadence

Other stuff

brew install p7zip brew install tree brew cask install wireshark brew install vim brew install graphviz brew install jq

Basics

Browsers

  • Chrome
    • brew cask install google-chrome
    • setup extensions

Communication & Storage Tools

  • Dropbox
    • brew cask install dropbox
  • Zoom Client for Meetings
  • Google Drive
    • brew cask install google-drive
  • FaceTime
  • Box
  • Slack
    • brew cask install slack
    • setup accounts
  • Fuze
    • install fuze chrome extension
  • Skype
    • brew cask install skype

Text Editors

  • Visual Studio Code
brew cask install visual-studio-code
  • Sublime Text 3

    • plug-ins : Package Control
    • create command line shortcut
    ln -s /Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl /usr/local/bin/sublime

Shell Utilities

  • autojump

    • brew install autojump
    • add to .bashrc
    [[ -s $(brew --prefix)/etc/autojump.sh ]] && . $(brew --prefix)/etc/autojump.sh
  • bash-completions

    • brew install bash-completions
    • add to .bashrc
    if [ -f $(brew --prefix)/etc/bash_completion ]; then
        . $(brew --prefix)/etc/bash_completion
    fi
  • fzf

brew install fzf

then run the installer.

.bashrc should look like this

# fzf - fuzzy finder

[ -f ~/.fzf.bash ] && source ~/.fzf.bash
export FZF_DEFAULT_OPTS="--extended"

Use CTRL-r for history, CTRL-t for finding files, put **[TAB] in any command for completions.

  • fswatch

Virtualization & Containers

  • brew cask install virtualbox (is this needed?)
  • brew install kubectl
  • Install Docker for Mac (Edge), and configure Kubernetes

See Romin Irani's Blog for more details

  • setup docker-registry, if needed
  • verify anti-virus networking setup (e.g. symantec network threat protect)?

Dev Environments

Set up CA using CFSSL

Also see docker-kubernetes-tls-guide

# needs go installed first

go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson

Envoy

https://github.com/envoyproxy/envoy/blob/master/bazel/README.md

brew install coreutils # for realpath
brew install wget
brew install cmake
brew install libtool
brew install go
brew install java8
brew install bazel
brew install automake

Web Development

  • brew install node

Go

  • brew install go
  • in .profile add
export GOPATH=~/dev/go
export PATH=$PATH:$GOPATH/bin
brew install dep

For go 1.10: go get -u golang.org/x/vgo

  • vscode : install go extension

proto3 and grpc

Protobuf

# install protoc compiler - needed for all protobuf work
brew install protobuf

or, do it the hard way

mkdir tmp
cd tmp
git clone https://github.com/google/protobuf
cd protobuf
./autogen.sh
./configure
make
make check
sudo make install

Language specific generators and validators

# golang
go get -u github.com/golang/protobuf/protoc-gen-go
# go validators
go get -u github.com/golang/protobuf/protoc-gen-go

GRPC Gateway

# TODO - check
go get google.golang.org/grpc
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger

Curl with HTTP2 support

# only if locl curl does not support http2
brew install curl --with-nghttp2

Mobile Development

Flutter

Follow Flutter Setup for MacOS

git clone -b dev https://github.com/flutter/flutter.git

Android

Download Android Studio

XCode

# Install XCode from app store.

# Ensure xcode command line tools are using installed version of XCode

sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer

# Enable XCode license
sudo xcodebuild -license

# Enable Developer mode (allows debugging)
sudo DevToolsSecurity -enable

# Run iOS Simulator
open -a Simulator

Other dependencies

brew install --HEAD libimobiledevice
brew install ideviceinstaller
brew install ios-deploy
brew install cocoapods
pod setup

python3

brew install python3

Rust

Install via rustup

Fix .bash_profile / .profile to remove redundant path settings

https setup

inspired by how to set up stress free ssl on os x

  • dnsmasq
brew install dnsmasq
mkdir -pv $(brew --prefix)/etc
sudo cp -v $(brew --prefix dnsmasq)/homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons
sudo launchctl load -w /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
sudo mkdir -pv /etc/resolver

Debugging & General Dev Tools

  • Wireshark
    • brew cask install wireshark

Android Development

Android Studio

Python

  • brew install python (includes setuptools and pip)
  • pip install virtualenv
  • pip install --upgrade pip
  • brew install python3
  • use built in venv for python3

TODO : scipy, jupyter

Java

download & install java from oracle site

To uninstall java plugin :

sudo rm -rf JavaControlPanel.prefPane

sudo rm -rf JavaAppletPlugin.plugin/

##Launchd setup

From https://discussions.apple.com/thread/2781309?start=0&tstart=0 Note: -w is probably deprecated as well

setting kernel variables permanently on startup vm.shared_region_unnest_logging=0 to avoid chrome errors

sudo -s
cat > /Library/LaunchDaemons/sysctl.plist << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>sysctl</string>
 <key>ProgramArguments</key>
 <array>
 <string>/usr/sbin/sysctl</string>
 <string>-w</string>
 <string>vm.shared_region_unnest_logging=0</string>
 </array>
 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>
EOF
launchctl load /Library/LaunchDaemons/sysctl.plist

##Mac Debugging / Fixing

#show kernel extensions
kextstat -kl | awk ' !/apple/ {print $6 $7} '

#show launch agents/daemons global
sudo launchctl list | sed 1d | awk ' !/apple/ {print  $3 }'

#show launch agents/daemons for user
launchctl list | sed 1d | awk ' !/apple/ {print  $3 }'

#list library components
ls -1A {~/,/}Library/{Ad,Compon,Ex,Fram,In,La,Mail/Bu,P*P,Priv,Qu,Scripti,Sta}* 2> /dev/null

finding the plist file for a process

function launchfind() {
    local LaunchctlPATHS=( \
        ~/Library/LaunchAgents \
        /Library/LaunchAgents \
        /Library/LaunchDaemons \
        /System/Library/LaunchAgents \
        /System/Library/LaunchDaemons \
    )
    for curPATH in "${LaunchctlPATHS[@]}"; do
        grep -r "$curPATH" -e "$1"
    done
}

Media Tools

  • GIMP
    • brew cask install gimp

Remote Desktop Tools

GravitDesigner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.