Skip to content

Instantly share code, notes, and snippets.

@nabinno

nabinno/160121_gcpja.md

Last active January 22, 2016 04:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nabinno/bcec659a0e7f741b2327 to your computer and use it in GitHub Desktop.
Save nabinno/bcec659a0e7f741b2327 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
160121_gcpja.md

Table of Contents

Borg of GKE

About Kubernetes

About Ian Lewis

  1. Developer Advocate / Support

Background

  1. Services

    1. Gmail 400M people

    2. Youtube 6B movies

  2. Infrastructure

    1. Faster Project with KDDI in 2014

      1. 38 countries, edge location (E2E)

      2. MapReduce, BigQuery

Anout Container

  1. LXC developed by Google, etc.

  2. OS-level virtualization

  3. Pros

    1. Light

    2. Portable

    3. Efficiency

Anout Borg

  1. Configuration file

  2. 100K tasks

    > process CPU
    > Database

  3. Google not use Docker but LXC

    1. Borg is container scheduler

    2. Borg developers are Kubernetes developers

  4. Functions

    1. Cluster management

      1. Architecture

        1. Pod

          1. Containers

            1. Services of application and so on

          2. Labels

            1. Frontendlabel

            2. Backend label

      2. Replication Controller

        1. Fail safe

        2. Hot deployment

        3. Scale up

        4. Scale down

      3. LB

      4. Service of connection both of frontend and backend

        1. e.g.

          1. Virtual IP to backend Pod

            >

          2. Configuration file

            1. Kubernetes configuration file is YAML

            2. Borg configuration file is Original

          3. kubectl

Takusuta by CyberAgent

AWS

GKE

Price

12K per month

Free until 5 nodes

Pod

kubectl

  1. Replication controller (RC)

    1. scheduler

    2. configuration file

      1. yaml

      2. json

  2. Service

    1. IP: Layer 4

    2. Round Robin

    3. Load balancer

      1. Global IP = GCE network address

    4. Certification

      1. not ELB

      2. node Pod

      3. GCE-LB

      4. Cross egion LB

      5. Select region/zone

        1. attend Generation

Create Cluster

  1. scope

  2. gcloud describe

HTTPS LB

Multi Container Pod

  1. Fluentd -> bigquery

Single Container Pod

  1. Default backend of GCE-LB

Config sample

Dockerfile

RC config file

Service

CI

Upload to GCR

Travis to Jenkins on GCE

Rolling update RC

  1. make Unix time

Scale up Cluster / RC

Log management & analytics

Clou logging

  1. GPE

Cloud monitoring

  1. Stack logging

  2. Slack and mail notification

Kubedash

  1. awesome!

  2. kubectl create f

Arakawa at Recruit life stle

Profile

iOS

Backend

Project background

IOT

Sensor devise

Hard DevOps

Architecture

Google Cloud Storage = Amazon S3

Proxy Server

  1. HTTP LB (L7LB)

    1. Functions

      1. 1M eq per sec

      2. Cross egion LB

      3. SSL termination

      4. HTTP/2

    2. Setup by CLI

  2. H2O proxy –> L7LB

  3. Cube Proxy

CI

  1. Github –> CircleCI -Rolling update-> GCE –> Slack

GKE and Kubernetes

Log management & analytics

  1. Cloud logging

    1. Google Cloud Monitoring

    2. Stackdriver

    3. Slack

Data analysis

  1. Problems

    1. Define and plan

    2. Save

    3. Analysis

  2. BigQuery

    1. Not Real-time

    2. Price

      1. 11 USD per month for BigQuery

      2. 7 USD per month for GCE

  3. Fluentd –> BigQuery

  4. Case example of BigQuery

    1. SQL-like

    2. Sensor around Himejijo

    3. 24M record

    4. 6.6 sec

    5. 1.43 GB

    6. Batch

Microsevrvice

Autoscaling in Voyage Group

Profile

Misa Kondo

Voyage Group

DevOps

Fluct (SSP of Online Advertising)

  1. Impression messurement

CDN

Load Balancing

HTTPS LB (L7LB)

  1. HTTP load balancing serving capacity

    1. 7500 RPS (request per second) / 1 instance

Autoscaler

Make as interface for AWS and GCP

Configuration

  1. Instance Group

    1. Instance Template

Case example

  1. DNS –> Route53 –>

Cloud Monitoring

CDU average

Cloud Logging

Log management & analytics

  1. BigQuery

  2. Case example

    1. Access log in Nginx

      1. UserAgent

      2. Impression messurement

Data mining in Voyage Group

Profile

Voyage Group

Zucks Scientist

Display Advertising

Machine Learning

Pre processing

Research algorithm

BigQuery

Predictive analytics (KPI)

CTR

CVR

Price

14K USD per 73TB, Month

String insert

  1. High expense

Log management & analytics

Redash

Datalab

  1. Price 11 USD per month
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment