LBaaS in OpenContrail

lbaas.txt

Check for package installation:

root@oblocknode04:/etc/contrail# dpkg -l | grep haproxy
ii  haproxy                              1.5.4-1ppa1~precise                   amd64        fast and reliable load balancing reverse proxy
root@oblocknode04:/etc/contrail# dpkg -l | grep iproute
ii  iproute                              1:3.12.0-2                            all          transitional dummy package for iproute2
ii  iproute2                             3.12.0-2                              amd64        networking and traffic control tools
root@oblocknode04:/etc/contrail# dpkg -l | grep netns
ii  python-opencontrail-vrouter-netns    2.21.2-28                             amd64        OpenContrail vrouter network namespace package

In testbed.py specify
env.enable_lbaas = True

Workflow:

- Create a VIP network

    - neutron net-create vipnet
    - neutron subnet-create —name vipsubnet vipnet 2.2.2.0/24
- Create a pool network

    - neutron net-create poolnet
    - neutron subnet-create —name poolsubnet polite 3.3.3.0/24
- Create a pool for HTTP

    - neutron lb-pool-create —lb-method ROUND_ROBIN —name test pool —protocol HTTP —subnet-id poolsubnet
- Add members to the pool

    - neutron lb-member-create —address 3.3.3.2 —protocol-port 80 testpool
    - neutron lb-member-create —address 3.3.3.3 —protocol-port 80 testpool
- Create a VIP for HTTP and associate it to the testpool

    - neutron lb-vip-create —name testvip —protocol-port 80 —protocol HTTP —subnet-id vipsubnet testpool
- Associating a FIP to a VIP 😃


    - Similar to any other floatingip-associate

        - neutron floatingip-associate <vip-uuiid> <floatingip-uuid>
- Create healthmonitor

    - neutron lb-healthmonitor-create —delay 20 —timeout 10 —max-retries 3 —type HTTP
- Associate healthmonitor

    - neutron lb-healthmonitor-associate <healthmonitor-uuid> testpool
- For SSL

    - Manually copy ssl-cert.pem to compute (or all computes). We only support single cert today.
    - Update /etc/contrail/contrail-vrouter-agent.conf

            - haproxy_ssl_cert_path=<path to ssl-cert.pem copied in step above>
    - Restart vrouter-agent
    - neutron lb-vip-create —name testvip —protocol-port 443 —protocol HTTP —subnet-id vipsubnet testpool

Scheduling options

- Round Robin: go in turns according to set weight.
- Least connection: pool members with least connections are preferred.
- Source ip: source ip hash based scheme

Pool members liveness check

- TCP: LB makes a tcp connection
- HTTP: LB makes HTTP request on the established TCP connection
- PING: LB pings pool members

Neutron lbaas commands: https://wiki.openstack.org/wiki/Neutron/LBaaS/CLI

  lb-healthmonitor-create        Create a health monitor
  lb-healthmonitor-delete        Delete a given health monitor
  lb-healthmonitor-list          List health monitors that belong to a given tenant
  lb-healthmonitor-show          Show information of a given health monitor
  lb-healthmonitor-update        Update health monitor's information

  lb-healthmonitor-associate     Associate a health monitor with a pool
  lb-healthmonitor-disassociate  Disassociate a health monitor with a pool

  lb-member-create               Create a member
  lb-member-delete               Delete a given member
  lb-member-list                 List members that belong to a given tenant
  lb-member-show                 Show information of a given member
  lb-member-update               Update member's information

  lb-pool-create                 Create a pool
  lb-pool-delete                 Delete a given pool
  lb-pool-list                   List pools that belong to a given tenant
  lb-pool-show                   Show information of a given pool
  lb-pool-update                 Update pool's information

  lb-pool-stats                  Retrieve traffic statistics for pool

  lb-vip-create                  Create a vip
  lb-vip-delete                  Delete a given vip
  lb-vip-list                    List vips that belong to a given tenant
  lb-vip-show                    Show information of a given vip
  lb-vip-update                  Update vip's information

Config files:

Set log_level to SYS_DEBUG in contrail-svc-monitor.conf, contrail-api.conf, contrail-schema.conf and contrail-svc-monitor.conf to see transactional records for the LB operations above.

Multiple availability zones can be specified here as availability_zone

contrail-svc-monitor.conf
[DEFAULTS]
ifmap_server_ip=172.16.80.4
ifmap_server_port=8443
ifmap_username=svc-monitor
ifmap_password=svc-monitor
api_server_ip=172.16.80.4
api_server_port=8082
api_server_use_ssl=False
zk_server_ip=172.16.80.4:2181
log_file=/var/log/contrail/contrail-svc-monitor.log
cassandra_server_list=172.16.80.4:9160
disc_server_ip=172.16.80.4
disc_server_port=5998
region_name=RegionOne
log_local=1
log_level=SYS_NOTICE
rabbit_server=172.16.80.4
rabbit_port=5672

[SECURITY]
use_certs=False
keyfile=/etc/contrail/ssl/private_keys/svc_monitor_key.pem
certfile=/etc/contrail/ssl/certs/svc_monitor.pem
ca_certs=/etc/contrail/ssl/certs/ca.pem

[SCHEDULER]
analytics_server_ip=172.16.80.4
analytics_server_port=8081

On the vrouter: look at /etc/contrail/contrail-vrouter-agent.conf

[SERVICE-INSTANCE]
# Path to the script which handles the netns commands
netns_command=/usr/bin/opencontrail-vrouter-netns

# Number of workers that will be used to start netns commands
#netns_workers=1

# Timeout for each netns command, when the timeout is reached, the netns
# command is killed.
#netns_timeout=30

Logs:
/var/log/contrail/contrail-api.log
/var/log/contrail/contrail-api-0-stdout.log
/var/log/contrail/api-0-zk.log

/var/log/contrail/schema-zk.log
/var/log/contrail/contrail-schema.log
/var/log/contrail/contrail-schema-stdout.log

/var/log/contrail/contrail-svc-monitor.log
/var/log/contrail/contrail-svc-monitor-stdout.log
/var/log/contrail/svc-monitor.err
/var/log/contrail/svc-monitor-zk.log

LB Configs:

- ip netns list
- Check /var/lib/contrail/loadbalancer/<uuid>/etc/haproxy/haproxy.cfg

Introspect:
http://10.87.25.250:8088/Snh_ServiceInstanceList?si_name=
http://10.87.25.250:8082/loadbalancer-pools
http://10.87.25.250:8081/analytics/uves/service-instances

Issues:

Issue 1: A vip does not launch any haproxy instance.

    - Look at contrail-api log and schema log first
    - Is the svc_monitor process stuck or hung, look for its logs
    - Restart contrail_svc_monitor, see if haproxy instance got created correctly.

-----------------------------

Issue 2: Killing haproxy on hypervisor and see it it spawns a new haproxy somewhere else, but I don't see that happening.

- Check haproxy process on vrouter "ps -ef | grep haproxy"
- Check contrail config for VM back-refs pointing to the old haproxy instance.
- The database still points to the old active/standby instances which does not seem to be correct.

-----------------------------

Issue 3: Multiple haproxy processes getting spawned for a single LBaaS.

Has been observed when LB config is updated. Old haproxy handles old sessions and new one is spawned for new sessions and new configs.

-----------------------------

Issue 4: SG applied for VIP must apply for the SI VM as well. Check that.

Other tools:

- Run contrail-svc-monitor manually

    - ps aux | grep contrail-svc-monitor
contrail  2320  0.0  0.0 311980 60364 ?        Sl   00:59   0:36 /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
    - root     13181  0.0  0.0  10472  2196 pts/25   S+   12:17   0:00 grep --color=auto contrail-svc-monitor
    - See where it crashes

        - /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
- tcpdump on the corresponding veth interfaces for data packet flow
- echo "show stat" | nc -U 1d663edb-cab2-45bd-a041-4eb019695ba1.haproxy.cfg.sock

_________________________

Details of Implementation:

You can read more about how it exactly works at  /usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py

-----------------------------