Skip to content

Instantly share code, notes, and snippets.

@harryge00

harryge00/Istio环境部署说明.md

Last active July 4, 2019 10:15
Show Gist options
  • Save harryge00/c6159d7c6b1382d7af236cebeef1cb77 to your computer and use it in GitHub Desktop.
Save harryge00/c6159d7c6b1382d7af236cebeef1cb77 to your computer and use it in GitHub Desktop.
Download ZIP
mesos-dns 与 CNI 集成
Raw
Istio环境部署说明.md

Mesos-DNS 部署

Mesos-dns是 mesos 服务发现工具，能查找app的Ip，端口号以及master，leader等信息。在Apache Mesos 集群中,允许应用程序和服务通过域名系统(DNS)来相互定位。Mesos-DNS 的特点是轻量、无状态,易于部署和维护。Mesos-DNS 将每个集群中正在运行的应用程序的域名转换成IP 地址和端口号。这使得任何链接到集群中运行的服务都可以通过DNS 进行查找。 Mesos-DNS是简单和无状态的，它不需要一致性机制，持久存储或日志复制，因为应用所需的心跳，健康监控或生命周期管理等功能都已经由Master、Agent和框架所提供。Mesos-DNS可以通过使用像Marathon这样的框架来实现容错，可以监控应用程序运行状况并在应用出现故障时重新启动它。Mesos-DNS因故障重新启动时，无需进一步协调它会从Master节点检索最新的应用状态，然后处理DNS请求。可以通过增加Master节点，即可提高可用性并为大规模集群中的DNS请求提供负载均衡。

Mesos-DNS 安装

Mesos-DNS 可以装在Mesos集群的任意一台机器上，或者是在同一网络的一台专用机器上。为了实现HA，可以在多个master上分别部署mesos-dns 实例。

wget https://github.com/mesosphere/mesos-dns/releases/download/v0.6.0/mesos-dns-v0.6.0-linux-amd64
sudo mv mesos-dns-v0.6.0-linux-amd64 /usr/local/bin/

接下来，按照链接 http://mesosphere.github.io/mesos-dns/docs/configuration-parameters.html 说明为你的集群创建一个配置文件。可以这样启动Mesos-DNS

sudo su
mkdir /etc/mesos-dns
cat > /etc/mesos-dns/config.json << EOF
{
  "zk": "zk://10.101.160.15:2181/mesos",
  "masters": ["10.101.160.15:5050", "10.101.160.16:5050", "10.101.160.17:5050"],
  "refreshSeconds": 60,
  "mesosCredentials": {
    "principal": "admin",
    "secret": "password"
  },
  "mesosAuthentication": "basic",
  "ttl": 60,
  "domain": "mesos",
  "port": 53,
  "resolvers": ["114.114.114.114"],
  "timeout": 5,
  "email": "root.mesos-dns.mesos"
}
EOF

# Create Systemd service
cat > /usr/lib/systemd/system/mesos-dns.service << EOF
[Unit]
Description=Mesos-DNS
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/local/bin/mesos-dns -config=/etc/mesos-dns/config.json
Restart=on-failure
RestartSec=20

[Install]
WantedBy=multi-user.target
EOF

# Start mesos-dns 
systemctl daemon-reload
systemctl start mesos-dns

设置从节点

允许Mesos任务使用Mesos-DNS作为主DNS服务，必须在每个从节点上修改文件 /etc/resolv.conf增加新的nameserver。例如，如果多个mesos-dns运行的服务器地址是10.181.64.11，10.181.64.12，10.181.64.13， 则应该在每个从节点/etc/resolv.conf开始加上三行nameserver 10.181.64.1X。可以通过运行下面的命令实现：

sudo sed -i '1s/^/nameserver 10.181.64.11\n /' /etc/resolv.conf
sudo sed -i '2s/^/nameserver 10.181.64.12\n /' /etc/resolv.conf
sudo sed -i '3s/^/nameserver 10.181.64.13\n /' /etc/resolv.conf

这些条目的顺序将确定从节点连接Mesos-DNS实例的顺序。可以通过设置options rotate在nameserver之间选择负载均衡的轮循机制, 如

# cat /etc/resolv.conf 
options rotate
nameserver 10.181.64.11
nameserver 10.181.64.12
nameserver 10.181.64.13

验证mesos-dns

我们可以在 marathon 上创建1个应用，比如nginx，然后在从节点应该能获取到对应的dns解析记录

nslookup nginx.marathon.slave.mesos
Server:		172.31.23.220
Address:	172.31.23.220#53

Name:	nginx.marathon.slave.mesos
Address: 172.31.27.226

CNI 集成

Mesos 源码中已经有Mesos CNI: https://github.com/apache/mesos/tree/master/src/slave/containerizer/mesos/isolators/network/cni 所以不管是rpm安装或者编译，都会产生cni插件的目录。rpm安装的方式，CNI插件在 /usr/libexec/mesos

$ ls /usr/libexec/mesos
mesos-cni-port-mapper  mesos-default-executor  mesos-executor  mesos-io-switchboard    mesos-tcp-connect
mesos-containerizer    mesos-docker-executor   mesos-fetcher   mesos-logrotate-logger  mesos-usage

但是这些插件依赖于 https://github.com/containernetworking/plugins 官方的二进制。所以我们需要在从节点添加来自于containernetworking的二进制文件，并在 mesos-slave 配置中指定。

wget https://github.com/containernetworking/plugins/releases/download/v0.7.4/cni-plugins-amd64-v0.7.4.tgz
tar -xzvf cni-plugins-amd64-v0.7.4.tgz -C /etc/cni
mkdir /etc/cniconf
cat > /etc/cniconf/ucr-default-bridge.conf << EOF
{
  "name": "mesos-bridge",
  "type" : "mesos-cni-port-mapper",
  "excludeDevices" : ["ucr-br0"],
  "chain": "UCR-DEFAULT-BRIDGE",
  "delegate": {
    "type": "bridge",
    "bridge": "ucr-br0",
    "isDefaultGateway": true,
    "forceAddress": false,
    "ipMasq": true,
    "hairpinMode": true,
    "ipam": {
      "type": "host-local",
      "subnet": "172.31.254.0/24",
      "routes": [
      { "dst":
        "0.0.0.0/0" }
      ]
    }
  }
}
EOF

配置mesos-slave

echo "MESOS_NETWORK_CNI_CONFIG_DIR=/etc/cniconf" >>  /etc/default/mesos-slave
echo "MESOS_NETWORK_CNI_PLUGINS_DIR=/usr/libexec/mesos:/etc/cni" >>  /etc/default/mesos-slave
systemctl daemon-reload
systemctl restart mesos-slave

安装Istio 组件

分别部署etcd, apiserver, zipkin，以应用的形式： etcd

{
  "id": "/etcd",
  "cmd": "/usr/local/bin/etcd -advertise-client-urls=http://0.0.0.0:2379 -listen-client-urls=http://0.0.0.0:2379",
  "cpus": 0.5,
  "mem": 512,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "docker": {
      "forcePullImage": false,
      "image": "quay.io/coreos/etcd:latest",
      "parameters": [],
      "privileged": false
    },
    "volumes": [],
    "portMappings": [
      {
        "containerPort": 2379,
        "hostPort": 31379,
        "labels": {},
        "name": "etcd",
        "protocol": "tcp",
        "servicePort": 10000
      }
    ]
  },
  "env": {
    "SERVICE_IGNORE": "1"
  },
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "healthChecks": [
    {
      "protocol": "HTTP",
      "path": "/health",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3
    }
  ],
  "labels": {
    "consul": ""
  }
}

apiserver

{
  "id": "/apiserver",
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "gcr.io/google_containers/kube-apiserver-amd64:v1.7.3"
    },
    "portMappings": [
      {
        "containerPort": 8080,
        "hostPort": 30080,
        "labels": {
          "VIP_0": "/apiserver:10080"
        },
        "protocol": "tcp"
      }
    ]
  },
  "cpus": 0.3,
  "mem": 640,
  "requirePorts": false,
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "healthChecks": [],
  "fetch": [],
  "constraints": [],
  "cmd": "kube-apiserver --etcd-servers http://etcd.marathon.slave.mesos:31379 --service-cluster-ip-range 10.99.0.0/16 --insecure-port 8080 -v 2 --insecure-bind-address 0.0.0.0",
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_VHOST": ""
  }
}

zipkin

{
  "id": "/zipkin",
  "cmd": null,
  "cpus": 1,
  "mem": 1024,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "docker": {
      "forcePullImage": false,
      "image": "docker.io/openzipkin/zipkin:2.7",
      "parameters": [],
      "privileged": false
    },
    "volumes": [],
    "portMappings": [
      {
        "containerPort": 9411,
        "hostPort": 31767,
        "labels": {},
        "name": "zipkinport",
        "protocol": "tcp",
        "servicePort": 10005
      }
    ]
  },
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_VHOST": "",
    "consul": ""
  },
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "healthChecks": [
    {
      "protocol": "HTTP",
      "path": "/health",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3
    }
  ]
}

Istio 可以在 master 上部署

cat > kubeconfig << EOF
apiVersion: v1
clusters:
- cluster:
    server: http://apiserver.marathon.slave.mesos:31080
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}

docker run -d -p 15010:15010 -p 15007:15007 -p 9093:9093 -v $PWD:/mnt/mesos/sandbox/ registry.cn-hangzhou.aliyuncs.com/zmcc-istio/pilot:91b8c8ab79238949756341c8493251f9d6313021  discovery --httpAddr :15007 --registries Mesos --domain marathon.slave.mesos --mesosVIPDomain marathon.slave.mesos --mesosMaster http://master.mesos:8080 --kubeconfig /mnt/mesos/sandbox/kubeconfig

部署Pod应用

Istio 部署成功后，即可部署应用。对于需要加入服务网格的服务，需要添加1个 pilot-agent 容器做流量劫持。1个pod例子如下：

{
  "environment": {
    "SERVICES_DOMAIN": "marathon.slave.mesos",
    "ENVOY_DEBUG": "debug"
  },
  "labels": {
    "ISTIO_SERVICE_NAME": "reviews",  # 希望使用的服务名，可以和ID不相同
    "version": "v1"            # 版本
  },
  "id": "/reviews",
  "containers": [
    {
      "name": "reviews",
      "resources": {
        "cpus": 0.2,
        "mem": 1024,
        "disk": 0,
        "gpus": 0
      },
      "endpoints": [
        {
          "name": "http-9080",
          "containerPort": 9080,
          "hostPort": 0,
          "protocol": [
            "tcp"
          ]
        }
      ],
      "image": {
        "kind": "DOCKER",
        "id": "istio/examples-bookinfo-reviews-v1:1.8.0"
      }
    },
    {
      "name": "proxy",
      "resources": {
        "cpus": 0.2,
        "mem": 512,
        "disk": 0,
        "gpus": 0
      },
      "exec": {
        "command": {
          "shell": "chmod +777 /dev/stdout; /usr/local/bin/istio-iptables.sh -p 15001 -u 1337 -m REDIRECT -i '*' -b 9080; iptables -t nat -I ISTIO_OUTPUT -d 172.31.0.0/16 -p tcp -m tcp --dport 5051 -j RETURN; su istio-proxy -c \"/usr/local/bin/pilot-agent proxy --discoveryAddress master.mesos:15010  --domain marathon.slave.mesos --serviceregistry Mesos --serviceCluster reviews-v1 --zipkinAddress zipkin.marathon.slave.mesos:31767 --configPath /var/lib/istio\""
        }
      },
      "image": {
        "kind": "DOCKER",
        "id": "registry.cn-hangzhou.aliyuncs.com/zmcc-istio/proxy_debug:mesos"
      }
    }
  ],
  "networks": [
    {
      "name": "mesos-bridge",
      "mode": "container"
    }
  ],
  "scaling": {
    "instances": 1,
    "kind": "fixed"
  },
  "scheduling": {
    "backoff": {
      "backoff": 1,
      "backoffFactor": 1.15,
      "maxLaunchDelay": 3600
    },
    "upgrade": {
      "minimumHealthCapacity": 1,
      "maximumOverCapacity": 1
    },
    "killSelection": "YOUNGEST_FIRST",
    "unreachableStrategy": {
      "inactiveAfterSeconds": 0,
      "expungeAfterSeconds": 0
    },
    "placement": {
      "constraints": []
    }
  },
  "executorResources": {
    "cpus": 0.1,
    "mem": 32,
    "disk": 10
  },
  "volumes": [],
  "fetch": []
}

pilot-agent 的启动脚本主要有两部分：

  1. 配置iptables规则将流量转发到15001端口，也就是envoy代理的端口
 /usr/local/bin/istio-iptables.sh -p 15001 -u 1337 -m REDIRECT -i '*' -b 9080

istio-iptables.sh 是官方的脚本，-b 9080 说明从端口9080流入的流量将被转发给 envoy，根据应用本身暴露端口不同可以修改，多个端口用,隔开。-u 1337 是说 uid为1337的用户可以不经过envoy代理，也就是pilot-agent这个进程，它在和pilot-discovery通信的时候，不需要经过envoy。

  1. pilot-agent 启动，这部分就是pilot-agent生成 envoy的各种配置，并启动envoy。

部署bookinfo测试

bookinfo的全部例子的json，都可以在https://github.com/harryge00/istio-on-marathon-mesos/tree/master/bookinfo-mesos 找到

git clone https://github.com/harryge00/istio-on-marathon-mesos
cd istio-on-marathon-mesos/bookinfo-mesos
curl -d "@details.json" -X POST http://localhost:8080/v2/pods
curl -d "@ratings.json" -X POST http://localhost:8080/v2/pods
curl -d "@reviews.json" -X POST http://localhost:8080/v2/pods
curl -d "@reviews-v2.json" -X POST http://localhost:8080/v2/pods
curl -d "@reviews-v3.json" -X POST http://localhost:8080/v2/pods
curl -d "@productpage.json" -X POST http://localhost:8080/v2/pods

然后，就可以获取productpage的hostport，然后在浏览器访问啦。

同目录下的yaml是各种virtualservice规则，可以测试

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment