skseth/osx_setup.md

## osx_setup.md

      
    Raw
  

              osx_setup.md
            
          
    Setting up my OS X machine

First Things First


Enable FileVault (for encryption) - save recovery key in safe place
Sophos Anti-virus - update files
Office for Mac (if needed for project)

Setting Up a Profile

Based on this stackoverflow answer .bash_profile should be always
if [ -f ~/.profile ]; then
    source ~/.profile
fi

if [ -f ~/.bashrc ]; then
    source ~/.bashrc
fi
.profile should contain stuff you want in every shell

.bashrc should contain only bash-specific stuff for interactive use - command prompts, bash completion etc.
xcode / xcode command line tools

Needed for several of the following tools such as brew, python
xcode-select --install

You can also install xcode directly via safari from : https://developer.apple.com/download/more/. If you use Safari, you can resume downloads.
Repository Managers


Install brew as per homebrew instructions


brew tap homebrew/cask-versions - to be able to install versions of casks


in .profile
- export PATH=/usr/local/bin:/usr/local/sbin:$PATH
- export HOMEBREW_GITHUB_API_TOKEN=


Languages and Editors

Visual Studio Code

brew cask install visual-studio-code
Intellij IDEA

Install intellij idea CE / Ultimate as needed
git (default on mac)


ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
eval "$(ssh-agent -s)"
Add following file (~/.ssh/config)

Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa


ssh-add -K ~/.ssh/id_rsa


git config --global user.name ""


git config --global user.email ""


install GitLens extension for vscode


Java


JDK setup

brew cask install adoptopenjdk
brew tap AdoptOpenJDK/openjdk (if you want old versions)
brew cask install <version> # e.g. adoptopenjdk11, adoptopenjdk8
Use /usr/libexec/java_home -V to see list of JDKs
Add following to .profile :
export JAVA_HOME=`/usr/libexec/java_home -v 1.8`


install Java extension pack for vscode


brew install gradle


Go


brew install go
in .profile

export GOPATH=~/go
export PATH=$GOPATH/bin:$PATH


vscode : install go extension, and go tools needed by extension
delve : go get -u github.com/go-delve/delve/cmd/dlv

Rust


brew install rustup


rustup-init


in .profile


export PATH="$HOME/.cargo/bin:$PATH"


rustup component add rls rust-analysis rust-src # language server and related components
install rls extension in vscode

Python3


brew install python3


install xcode, sudo xcodebuild -license, sudo xcode-select --install


pip3 install virtualenv virtualenvwrapper


python3 -m pip install --user --upgrade setuptools wheel twine


Configurinng python3 to use specific SSL certificates is surprisingly complicated.
Create a file ~/.ca/ca-bundle.crt : e.g.
Download https://curl.haxx.se/ca/cacert.pem, then append ~/.ca/ca.local.crt
then, in .profile
export SSL_CERT_FILE=$HOME/.ca/ca-bundle.crt
export REQUESTS_CA_BUNDLE=$HOME/.ca/ca-bundle.crt


in .profile

# virtualenv and virtualenvwrapper
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
source /usr/local/bin/virtualenvwrapper.sh


mkvirtualenv py3cv4 -p python3 # p3cv4 is env name - you can replace it with your choice


pip install pylint (for vscode)


install python extension for vscode


Setup python packaging - see https://packaging.python.org/overview/, and checkout this tutorial : https://packaging.python.org/tutorials/packaging-projects/
Python3 - Nachine learning / computer vision

From macOS for deep learning with Python, TensorFlow, and Keras, with updates from installing open cv4
opencv

OpenCV dependencies :

brew install cmake pkg-config
brew install jpeg libpng libtiff openexr
brew install eigen tbb
brew install wget

Download and build opencv
cd ~
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.0.0.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.0.0.zip
unzip opencv.zip
unzip opencv_contrib.zip
mv opencv-4.0.0 opencv
mv opencv_contrib-4.0.0 opencv_contrib
cd ~/opencv
mkdir build
cd build
workon cv
cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D PYTHON3_LIBRARY=`python -c 'import subprocess ; import sys ; s = subprocess.check_output("python-config --configdir", shell=True).decode("utf-8").strip() ; (M, m) = sys.version_info[:2] ; print("{}/libpython{}.{}.dylib".format(s, M, m))'` \
    -D PYTHON3_INCLUDE_DIR=`python -c 'import distutils.sysconfig as s; print(s.get_python_inc())'` \
    -D PYTHON3_EXECUTABLE=$VIRTUAL_ENV/bin/python \
    -D BUILD_opencv_python2=OFF \
    -D BUILD_opencv_python3=ON \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D INSTALL_C_EXAMPLES=OFF \
    -D OPENCV_ENABLE_NONFREE=ON \
    -D BUILD_EXAMPLES=ON ..

make opencv 4

make -j4 // use make instead of make -j4 if any error occurs


install

sudo make install


link opencv into virtual environment

cd /usr/local/python/cv2/python-3.7 # this path depends on using python 3.7
sudo mv cv2.cpython-37m-darwin.so cv2.so # the 37m depends on usingh python 3.7
cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.so cv2.so

Verify opencv works

cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.so cv2.so
Keras & Tensorflow

pip install scipy pillow
pip install imutils h5py requests progressbar2
pip install scikit-learn scikit-image
pip install matplotlib
touch ~/.matplotlib/matplotlibrc
echo "backend: TkAgg" >> ~/.matplotlib/matplotlibrc

pip install tensorflow
pip install keras

Test keras setup:ß
$ python
>>> import keras
Using TensorFlow backend.
>>>

Node and NPM

brew install node
npm install typescript
npm install -g @angular/cli
npm install -g create-react-app
TBD: vscode setup for angular, react, typescript
flutter / dart / android studio

Flutter on macos

Install flutter
Add flutter/bin to path
install android studio
xcode

brew update
brew install --HEAD usbmuxd
brew link usbmuxd
brew install --HEAD libimobiledevice
brew install ideviceinstaller
brew install ios-deploy
brew install cocoapods
pod setup

Install flutter / dart extensions for Intellij, android studio, visual studio code
Scala

brew install scala
brew install sbt
vscode : install Scala syntax extension
intellij : scala plugin
C/C++

XCode
Ruby

Erlang

TODO
Virtual Machines and Orchestration Engines

Virtualbox

brew cask install virtualbox
Minikube

brew cask install minikube
Ansible

deactivate 
pip3 install ansible
CA and DNS

DNS

https://medium.com/@kharysharpe/automatic-local-domains-setting-up-dnsmasq-for-macos-high-sierra-using-homebrew-caf767157e43
https://banck.net/2018/12/using-dnsmasq-on-mac-os-for-local-development/
brew install dnsmasq
echo "listen-address=127.0.0.1" >> $(brew --prefix)/etc/dnsmasq.conf
echo "port=5354" >> $(brew --prefix)/etc/dnsmasq.conf
sudo brew services start dnsmasq

sudo mkdir -v /etc/resolver
sudo tee -a /etc/resolver/local >> EOF
port 5354
nameserver 127.0.0.1
EOF
scutil --dns
On the host, you should set the following when creating any vm :
VBoxManage modifyvm "<VM name>" --natdnshostresolver1 on
*.minikube.local

echo "address=/.minikube.local/192.168.64.11" >> $(brew --prefix)/etc/dnsmasq.conf
CA

See https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309
mkdir ~/.ca
cd ~/.ca

# generate ca private key
openssl genrsa -des3 -out rootCA.key 4096

# see the key - optional
openssl rsa -in ca.local.key -noout -text

# extract ca public key - optional
openssl rsa -in ca.local.key -pubout -out ca.local.pubkey 

# generate root CA Cert
openssl req -x509 -new -nodes -key ca.local.key -sha256 -days 1024 -out ca.local.crt

# Per domain (e.g. artifactory.local)
DOMAIN=artifactory.local
openssl genrsa -out $DOMAIN.key 2048

openssl req -new -sha256 \
    -key $DOMAIN.key \
    -subj "/C=IN/ST=KA/O=Myself/CN=$DOMAIN" \
    -reqexts SAN \
    -config <(cat /etc/ssl/openssl.cnf \
        <(printf "\n[SAN]\nsubjectAltName=DNS:$DOMAIN,DNS:*.$DOMAIN")) \
    -out $DOMAIN.csr

# check the request
openssl req -in $DOMAIN.csr -noout -text

# sign the csr

openssl x509 -req \
    -extfile <(printf "subjectAltName=DNS:$DOMAIN,DNS:*.$DOMAIN") \
    -days 365 -in $DOMAIN.csr \
    -sha256 \
    -CA ca.local.crt -CAkey ca.local.key -CAcreateserial \
    -out $DOMAIN.crt

# print contents of certificates
openssl x509 -in $DOMAIN.pem -text

# see certificate being served by a site
echo | openssl s_client -showcerts -connect kibana.host.local:443 | openssl x509 -inform pem -noout -text

nginx

brew install nginx
sudo brew service start nginx
NOTE: sudo is important. Without that nginx cannot be accessed from outside the host vm
Setup server proxies : SSL example at https://gist.github.com/shijij/54c9b21f26c08a15a70c182f03cb15b4
Also see :
https://stackoverflow.com/questions/16042647/whats-the-de-facto-standard-for-a-reverse-proxy-to-tell-the-backend-ssl-is-used
X-Forwarded-Protocol: https
X-Forwarded-Ssl: on
X-Url-Scheme: https
lsof -nP -i4TCP:443 | grep LISTEN
Unfortunately, on mac, this may not be enough for guest VMs to access nginx on your host machine.
First, the firewall may
Repositories

brew install artifactory
Nexus

Download nexus 3.0 tar.gz file from : https://help.sonatype.com/repomanager3
Note : As of May 2019, brew install nexus installs old version.
Untar the installation into $(brew --prefix)/opt/, and rename  to nexus.
To setup as a service, follow instructions here : https://help.sonatype.com/repomanager3/installation/run-as-a-service
Create following file : /Library/LaunchDaemons/om.sonatype.nexus.plist
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.sonatype.nexus</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/opt/nexus/nexus3xx/bin/nexus</string>
        <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>
sudo chown root:wheel /Library/LaunchDaemons/com.sonatype.nexus.plist
sudo chmod 644 /Library/LaunchDaemons/com.sonatype.nexus.plist
sudo launchctl load /Library/LaunchDaemons/com.sonatype.nexus.plist
To change the port nexus listens on, edit /usr/local/opt/nexus/sonatype-work/nexus3/etc/nexus.properties.
You can also set RUN_AS_USER setting in nexus startup script (as shown in the plist file).
run_as_user='<username>'

Setup nginx with ssl as a frontend for nexus (e.g. https://repos.host.local ).
Docker Repository

Setup 2 docker repos :
docker-private on 8101
docker-hub on 8102, as proxy for dockerhub
setup nginx ssl setup in front.
For minikube, copy the ca crt to docker dir :
cat ~/.ca/ca.local.crt | minikube ssh "sudo mkdir -p /etc/docker/certs.d/docker.host.local && sudo tee /etc/docker/certs.d/docker.host.local/ca.crt"

docker login docker.host.local # provide nexus userid, password

#do the same for dockerhub.host.local
Now try this :
docker pull dockerhub.host.local/busybox
docker tag dockerhub.host.local/busybox:latest docker.host.local/busybox:latest
docker push docker.host.local/busybox:latest
pypi

Follow instructions at https://help.sonatype.com/repomanager3/formats/pypi-repositories
Create pypi-proxy, pypi-hosted and pypi-group
On client machine :
create ~/.pip/pip.conf as follows :
[global]
index = http://localhost:8085/repository/pypi-all/pypi
index-url = http://localhost:8085/repository/pypi-all/simple

Run 'pip config list -v' to verify the file is read correctly.
For publishing with pypi, setup ~/.pypirc
[distutils]
index-servers = pypi
[pypi]
repository: http://localhost:8081/repository/pypi-hosted/
username: <userid>
password: <password>

It may work to leave out username/password and enter it while uploading.

maven
pypi
yum
apt
cargo

Athens (Go)
Cargo (Rust)
pub (dart)
General development

CA setup
DNS
Service development

ELK Stack

See https://logz.io/blog/elk-mac/.
brew install elasticsearch logstash kibana

Elasticsearch

brew services start elasticsearch # verify http://localhost:9200
Check elastic search health : http://localhost:9200/_cluster/health?pretty
If there is no response, there may be an error.
NOTE:
As of May 1, 2019 on Mac High Sierra, there was the following bug : https://discuss.elastic.co/t/elasticsearch-6-7-0-homebrew-install-macos-10-14-4-fails-to-run-error-cluster-name-elasticsearch-nathan-subdirectory-exists-in-data-paths/174747/3
To fix : rm -rf /usr/local/var/lib/elasticsearch/elasticsearch_/

Kibana

brew services start kibana
Fix the kibana config :
sudo vi /usr/local/etc/kibana/kibana.yml
uncomment following lines :
server.port: 5601

elasticsearch.hosts: ["http://localhost:9200"]
Check all is working : http://localhost:5601/status

Logstash

You can run in
Add a syslog pipeline :
sudo vim /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"] 
    index => "syslog-demo"
  }
  stdout { codec => rubydebug }
}

Prometheus

brew install prometheus
Gitlab

Proxying Gitlab : https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl
Also see : https://knowledge.rootknecht.net/integrate-gitlab-ce-docker-with-external-nginx-reverse-proxy
nginx['listen_port'] = 8081
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
Also, set https for registry, mattermost, pages
sudo gitlab-ci
MySQL

redis

cassandra

kafka

GRPC / OpenAPI

Istio

Envoy

Machine Learning development

Airflow / Cadence

Other stuff

brew install p7zip
brew install tree
brew cask install wireshark
brew install vim
brew install graphviz
brew install jq
Basics

Browsers


Chrome

brew cask install google-chrome
setup extensions


Communication & Storage Tools


Dropbox

brew cask install dropbox


Zoom Client for Meetings
Google Drive

brew cask install google-drive


FaceTime
Box
Slack

brew cask install slack
setup accounts


Fuze

install fuze chrome extension


Skype

brew cask install skype


Text Editors


Visual Studio Code

brew cask install visual-studio-code


Sublime Text 3

plug-ins : Package Control
create command line shortcut

ln -s /Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl /usr/local/bin/sublime


Shell Utilities


autojump

brew install autojump
add to .bashrc

[[ -s $(brew --prefix)/etc/autojump.sh ]] && . $(brew --prefix)/etc/autojump.sh


bash-completions

brew install bash-completions
add to .bashrc

if [ -f $(brew --prefix)/etc/bash_completion ]; then
    . $(brew --prefix)/etc/bash_completion
fi


fzf


brew install fzf

then run the installer.
.bashrc should look like this
# fzf - fuzzy finder

[ -f ~/.fzf.bash ] && source ~/.fzf.bash
export FZF_DEFAULT_OPTS="--extended"

Use CTRL-r for history, CTRL-t for finding files, put **[TAB] in any command for completions.

fswatch

Virtualization & Containers


brew cask install virtualbox (is this needed?)
brew install kubectl
Install Docker for Mac (Edge), and configure Kubernetes

See Romin Irani's Blog for more details

setup docker-registry, if needed
verify anti-virus networking setup (e.g. symantec network threat protect)?

Dev Environments

Set up CA using CFSSL

Also see docker-kubernetes-tls-guide
# needs go installed first

go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson
Envoy

https://github.com/envoyproxy/envoy/blob/master/bazel/README.md
brew install coreutils # for realpath
brew install wget
brew install cmake
brew install libtool
brew install go
brew install java8
brew install bazel
brew install automake

Web Development


brew install node

Go


brew install go
in .profile add

export GOPATH=~/dev/go
export PATH=$PATH:$GOPATH/bin
brew install dep

For go 1.10: go get -u golang.org/x/vgo

vscode : install go extension

proto3 and grpc

Protobuf

# install protoc compiler - needed for all protobuf work
brew install protobuf

or, do it the hard way
mkdir tmp
cd tmp
git clone https://github.com/google/protobuf
cd protobuf
./autogen.sh
./configure
make
make check
sudo make install

Language specific generators and validators

# golang
go get -u github.com/golang/protobuf/protoc-gen-go

# go validators
go get -u github.com/golang/protobuf/protoc-gen-go

GRPC Gateway

# TODO - check
go get google.golang.org/grpc

go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger

Curl with HTTP2 support

# only if locl curl does not support http2
brew install curl --with-nghttp2

Mobile Development

Flutter

Follow Flutter Setup for MacOS
git clone -b dev https://github.com/flutter/flutter.git

Android

Download Android Studio
XCode

# Install XCode from app store.

# Ensure xcode command line tools are using installed version of XCode

sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer

# Enable XCode license
sudo xcodebuild -license

# Enable Developer mode (allows debugging)
sudo DevToolsSecurity -enable

# Run iOS Simulator
open -a Simulator

Other dependencies

brew install --HEAD libimobiledevice
brew install ideviceinstaller
brew install ios-deploy
brew install cocoapods
pod setup

python3

brew install python3

Rust

Install via rustup
Fix .bash_profile / .profile to remove redundant path settings
https setup

inspired by how to set up stress free ssl on os x

dnsmasq

brew install dnsmasq
mkdir -pv $(brew --prefix)/etc
sudo cp -v $(brew --prefix dnsmasq)/homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons
sudo launchctl load -w /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
sudo mkdir -pv /etc/resolver
Debugging & General Dev Tools


Wireshark

brew cask install wireshark


Android Development

Android Studio
Python


brew install python (includes setuptools and pip)
pip install virtualenv
pip install --upgrade pip
brew install python3
use built in venv for python3

TODO : scipy, jupyter
Java

download & install java from oracle site
To uninstall java plugin :
sudo rm -rf JavaControlPanel.prefPane
sudo rm -rf JavaAppletPlugin.plugin/
##Launchd setup
From https://discussions.apple.com/thread/2781309?start=0&tstart=0
Note: -w is probably deprecated as well
setting kernel variables permanently on startup
vm.shared_region_unnest_logging=0 to avoid chrome errors
sudo -s
cat > /Library/LaunchDaemons/sysctl.plist << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>sysctl</string>
 <key>ProgramArguments</key>
 <array>
 <string>/usr/sbin/sysctl</string>
 <string>-w</string>
 <string>vm.shared_region_unnest_logging=0</string>
 </array>
 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>
EOF
launchctl load /Library/LaunchDaemons/sysctl.plist
##Mac Debugging / Fixing
#show kernel extensions
kextstat -kl | awk ' !/apple/ {print $6 $7} '

#show launch agents/daemons global
sudo launchctl list | sed 1d | awk ' !/apple/ {print  $3 }'

#show launch agents/daemons for user
launchctl list | sed 1d | awk ' !/apple/ {print  $3 }'

#list library components
ls -1A {~/,/}Library/{Ad,Compon,Ex,Fram,In,La,Mail/Bu,P*P,Priv,Qu,Scripti,Sta}* 2> /dev/null
finding the plist file for a process
function launchfind() {
    local LaunchctlPATHS=( \
        ~/Library/LaunchAgents \
        /Library/LaunchAgents \
        /Library/LaunchDaemons \
        /System/Library/LaunchAgents \
        /System/Library/LaunchDaemons \
    )
    for curPATH in "${LaunchctlPATHS[@]}"; do
        grep -r "$curPATH" -e "$1"
    done
}
Media Tools


GIMP

brew cask install gimp


Remote Desktop Tools


Teamviewer

GravitDesigner