Skip to content

Instantly share code, notes, and snippets.

@marccampbell

marccampbell/replicated-codeserver.md Secret

Created March 5, 2020 01:12
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save marccampbell/28585dfddafd25fe63be71e91cc0da0b to your computer and use it in GitHub Desktop.
Save marccampbell/28585dfddafd25fe63be71e91cc0da0b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
replicated-codeserver.md

Getting Started with your own CodeServer

There are some manual steps to get you started, but it should be quick. Let's dive in.

Add your SSH key to the project

  1. Log in to GCP and select the "Replicated Dev Environment" project
  2. Navigate to Compute Engine -> Metadata
  3. Add your public key to the ssh keys tab

Create an instance

  1. Log in to GCP and select the "Replicated Dev Environment" project
  2. Navigate to Compute Engine -> Images
  3. Select the newest codeserver-YYYY-MM-DD-nnnn image, and select "Create Instance"
  4. Name your instance repldev-<your-replicated-email-name> (e.g. repldev-marc or repldev-andrewl).
  5. Select the closest region to your location.
  6. Select a machine type (recommended n1-standard-8 : 8 CPUs, 30 GB Memory)

No other changes are necessary. Do not enable HTTP/HTTPS traffic.

This will create an instance with a 200 GB disk, with a most of the repl.dev environment already installed.

Click Create.

Create a home disk

Next we'll create a separate disk for home directories on your server. This is useful for instance upgrades. The codeserver image does get updated at times, and keeping your home directory on a separate disk will make the upgrade process very easy.

  1. Select your server from the list of servers
  2. Click Edit
  3. Click "Add New Disk"
  4. Name it <your-replicated-email>-home (e.g. marc-home or andrewl-home)
  5. Change the type to SSD
  6. Change the size to 500 GB
  7. Click Done
  8. Click Save

The above steps can also be achieved with the following gcloud commands.

EMAILNAME=<your-replicated-email-name>
GCPZONE=us-west2-a # change this if you're not in southern california
IMAGE="$(gcloud compute images list --project=replicated-dev-environment --filter="name~'codeserver-.+'" | awk 'END{print $1}')"

gcloud beta compute disks create ${EMAILNAME}-home \
    --project=replicated-dev-environment \
    --type=pd-ssd --size=500GB --zone=${GCPZONE} --physical-block-size=4096
gcloud compute instances create repldev-${EMAILNAME} \
    --project=replicated-dev-environment \
    --zone=${GCPZONE} --machine-type=n1-standard-8 --image=${IMAGE} --image-project=replicated-dev-environment \
    --subnet=default --network-tier=PREMIUM --maintenance-policy=MIGRATE \
    --boot-disk-size=200GB --boot-disk-type=pd-standard --boot-disk-device-name=repldev-${EMAILNAME} \
    --service-account=926355473660-compute@developer.gserviceaccount.com \
    --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append
gcloud compute instances attach-disk repldev-${EMAILNAME} \
    --project=replicated-dev-environment \
    --disk=${EMAILNAME}-home --zone=${GCPZONE}

Mount the disk

  1. SSH into your server by IP address

  2. Find the device name with lsblk (it's probably sdb, just find the 500 GB disk here)

    DEVICENAME=<device name>
sudo mkdir -p /mnt/home
sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/${DEVICENAME}
sudo mount -o discard,defaults /dev/${DEVICENAME} /mnt/home
sudo rsync -avHAX /home/ /mnt/home
sudo umount /mnt/home
sudo mount /dev/${DEVICENAME} /home
UUID="$(sudo blkid -s UUID -o value /dev/sdb)"
echo "UUID=${UUID} /home ext4 discard,defaults,nofail 0 2" | sudo tee -a /etc/fstab > /dev/null

Reboot for good measure. This will ensure the machine is working properly and all steps were completed successfully. (Reboot with sudo init 6). Note, the public IP will change after rebooting.

Do some things manually...

  1. SSH into your code server again by IP address
  2. Generate an ssh key ssh-keygen -t rsa -b 4096 -C "<replicated-email-address>"
  3. Add the ssh key to github
  4. sudo usermod -aG docker $USER
  5. sudo usermod -aG microk8s $USER
  6. Log out and log back in
  7. /usr/local/bin/startup-script.sh or $(which startup-script.sh)
  8. source ~/.bashrc

Connect to Cloudflare

  1. Configure Cloudflare cert & rules.

    Most devs do not have a cloudflare account, so you'll need an admin (Frank, Ethan, Marc, or Dmitriy) to ssh to your instance and complete the next few steps. Find a Cloudflare admin and share your dev instance ip with them, so that they can connect to your instance and complete the following:

    cloudflared login

    At this point the Cloudflare admin will be prompted to login and navigate to a URL. Once it's opened in the browser, select repl.dev as the authorized zone. If this is successful, the command above will terminate and display the location of a cert file.

    sudo cp <cert-file-path> /home/<developer-name>/.cloudflared/cert.pem
sudo chown <developer-name>:<developer-name> /home/<developer-name>/.cloudflared/cert.pem

    Now the Cloudflare admin should go to the Cloudflare 'repl.dev' project's 'Access' page, and add a new Access Policy for the developer, with the following settings:

    • Application Name:
    • Application Domain - subdomain:
    • Session Duration - 1 Month
    • Policies - Policy Name: Everyone
    • Policies - Decision: Allow
    • Include - Emails Ending In @replicated.com

    Now back to the developer for the rest of the steps

  2. Test that the cert works, by running:

    EMAILNAME=<your-replicated-email-name> # this is just the name, without "@replicated.com"
cloudflared tunnel --hostname ${EMAILNAME}.repl.dev --url ssh://localhost:22

    CTRL^C that.

    You should see output like this:

    $ sudo cloudflared tunnel --hostname ${EMAILNAME}.repl.dev --url ssh://localhost:22
INFO[0000] Version 2019.12.0                            
INFO[0000] GOOS: linux, GOVersion: go1.12.7, GoArch: amd64 
INFO[0000] Flags                                         hostname=<your-machine-name> logfile=/var/log/cloudflared.log proxy-dns-upstream="https://1.1.1.1/dns-query, https://1.0.0.1/dns-query" url="ssh://localhost:22"
INFO[0000] cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/ 
INFO[0000] Starting metrics server                       addr="127.0.0.1:36413"
INFO[0000] Proxying tunnel requests to http://127.0.0.1:34411 
INFO[0000] Connected to LAX                              connectionID=0
CTRL^C

  3. Install the cloudflared service:

    You can now ssh through cloudflare, but it requires a new port so that needs to stay on the IP address. In another window on the server:

    EMAILNAME=<your-replicated-email-name>
sudo mkdir /etc/cloudflared
sudo cp $HOME/.cloudflared/cert.pem /etc/cloudflared/
echo -e "hostname: ${EMAILNAME}.repl.dev\nurl: ssh://localhost:22\nlogfile: /var/log/cloudflared.log" | sudo tee -a /etc/cloudflared/config.yml > /dev/null
sudo cloudflared service install
sudo systemctl start cloudflared

  4. Tell iptables to allow connections (this does not persist between reboots):

    sudo iptables -P FORWARD ACCEPT  # https://github.com/ubuntu/microk8s/issues/75
sudo apt-get install -y iptables-persistent

    If the above command fails as follows:

    $ sudo apt-get install -y iptables-persistent
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

    Try first running the following in order to correct the issue:

    sudo apt-get install -f
sudo dpkg --configure -a

    TBD: resolve this in the original codeserver image

  5. Update microk8s:

    microk8s.reset
microk8s.enable dns ingress storage registry

Setup your mac & remote dev instance

  1. Install argo tunnel on your mac

    https://developers.cloudflare.com/argo-tunnel/downloads/

  2. Set up your local ssh config

    EMAILNAME=<your-replicated-email-name>
eval "echo \"$(cat ssh.config)\"" | tee -a $HOME/.ssh/config > /dev/null

    NOTE: you can clone this repo or download ssh.config file here

  3. Test ssh connections:

    ssh ${EMAILNAME}.repl.dev

  4. Login to docker hub on your remote instance (to cache auth)

    Create a DockerHub account if you don't already have one and ask an admin to invite you to the 'repldev' group. Run the following command on your remote instance:

    docker login

    This will create a ~/.docker/config.json file that will be used to authenticate requests to dockerhub during builds.

  5. Set your preferred editor on the remote instance

    select-editor

  6. Set up Visual Studio for Remote Development

    • Connect to your server ssh name.repl.dev to get the cloudflare login redirect going
    • Install the Visual Studio Remote SSH extension
    • Open a terminal and run the following on your remote instance:
    cd ~/go/src/github.com/replicatedhq
git clone git@github.com:replicatedhq/vandoor.git

  7. In Visual Studio open folder ~/go/src/github.com/replicatedhq/vandoor/

    • Open a terminal and run the following on your remote instance:
    make cache
skaffold dev

  8. TODO: Need to figure out NFS / Fuse mounts for Grayson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment