Hyperledger Fabric byfn docker-compose with fluent logging

00-HLF-logging-with-fluentd-README.md

What is this?

A sample docker compose file for BYFN from fabric-samples with fluentd log driver output enabled.

Also see Hyperledger Fabric fluent logging connected to EK stack for the same connected to EK stack (allowing Kibana view of Hyperledger Fabric logs).

NOTE The docker-compose file attached here is based on BYFN (fabric-samples) matching hyperledger fabric v1.1. However, it should be trivial to apply for others (such as v1.2).

The key config points are:

1. Adding deps and log driver config
2. fluentd docker container

To use

1. Update fabric-samples/first-network/docker-compose-cli.yaml with contents of docker-compose-cli-logging.yaml.
2. Place fluentd.conf under fabric-samples/first-network/fluentd/ directory (need to mkdir fluentd beforehand).
3. Launch byfn and start monitoring fluentd logs.
   > byfn.sh -m up; docker logs -f fluentd

---

To quit

1. Ctrl-c to exit docker logs.
2. Destroy byfn.
   > byfn.sh -m down

---

Example output

Starting with channel 'mychannel' and CLI timeout of '10'
proceeding ...
/home/vagrant/fabric/bin/cryptogen

##########################################################
##### Generate certificates using cryptogen tool #########
##########################################################
org1.example.com
org2.example.com

/home/vagrant/fabric/bin/configtxgen
##########################################################
#########  Generating Orderer Genesis block ##############
##########################################################

#################################################################
### Generating channel configuration transaction 'channel.tx' ###
#################################################################

#################################################################
#######    Generating anchor peer update for Org1MSP   ##########
#################################################################

#################################################################
#######    Generating anchor peer update for Org2MSP   ##########
#################################################################

Creating network "net_default" with the default driver
Creating network "net_byfn" with the default driver
Creating volume "net_peer0.org2.example.com" with default driver
Creating volume "net_peer1.org2.example.com" with default driver
Creating volume "net_peer1.org1.example.com" with default driver
Creating volume "net_peer0.org1.example.com" with default driver
Creating volume "net_orderer.example.com" with default driver
Creating fluentd
Creating orderer.example.com
Creating peer1.org1.example.com
Creating peer0.org2.example.com
Creating peer1.org2.example.com
Creating peer0.org1.example.com
Creating cli
2018-07-17 14:56:22 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2018-07-17 14:56:22 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    port 24224
    bind "0.0.0.0"
  </source>
  <match docker.**>
    @type stdout
  </match>
</ROOT>
2018-07-17 14:56:22 +0000 [info]: starting fluentd-1.2.3 pid=5 ruby="2.4.4"
2018-07-17 14:56:22 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluentd.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2018-07-17 14:56:23 +0000 [info]: gem 'fluentd' version '1.2.3'
2018-07-17 14:56:23 +0000 [info]: adding match pattern="docker.**" type="stdout"
2018-07-17 14:56:23 +0000 [info]: adding source type="forward"
2018-07-17 14:56:23 +0000 [info]: #0 starting fluentd worker pid=15 ppid=5 worker=0
2018-07-17 14:56:23 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2018-07-17 14:56:23 +0000 [info]: #0 fluentd worker is now running worker=0
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":" Version: 1.0.6"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":" Go version: go1.7.5"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"log":" OS/Arch: linux/amd64","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_name":"/peer0.org2.example.com","source":"stderr","log":" Chaincode:","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":"  Base Image Version: 0.3.2"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"log":"  Base Docker Namespace: hyperledger","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":"  Base Docker Label: org.hyperledger.fabric"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"log":"  Docker Namespace: hyperledger","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_name":"/peer0.org2.example.com","source":"stderr","log":"","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"log":"2018-07-17 14:56:23.507 UTC [ledgermgmt] initialize -> INFO 002\u001b[0m Initializing ledger mgmt","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"source":"stderr","log":"2018-07-17 14:56:23.509 UTC [kvledger] NewProvider -> INFO 003\u001b[0m Initializing ledger provider","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_name":"/peer0.org2.example.com","source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.509 UTC [kvledger.util] CreateDirIfMissing -> DEBU 004\u001b[0m CreateDirIfMissing [/var/hyperledger/production/ledgersData/ledgerProvider/]","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_name":"/peer0.org2.example.com","source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.509 UTC [kvledger.util] logDirStatus -> DEBU 005\u001b[0m Before creating dir - [/var/hyperledger/production/ledgersData/ledgerProvider/] does not exist","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.509 UTC [kvledger.util] logDirStatus -> DEBU 006\u001b[0m After creating dir - [/var/hyperledger/production/ledgersData/ledgerProvider/] exists","container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.512 UTC [kvledger.util] CreateDirIfMissing -> DEBU 007\u001b[0m CreateDirIfMissing [/var/hyperledger/production/ledgersData/chains/index/]"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.512 UTC [kvledger.util] logDirStatus -> DEBU 008\u001b[0m Before creating dir - [/var/hyperledger/production/ledgersData/chains/index/] does not exist"}
2018-07-17 14:56:23.000000000 +0000 docker.7217c904c5cf: {"container_id":"7217c904c5cf1252ee7278a84265ed2405784f81d494fddda3b7df7b3cf793ec","container_name":"/peer0.org2.example.com","source":"stderr","log":"\u001b[36m2018-07-17 14:56:23.512 UTC [kvledger.util] logDirStatus -> DEBU 009\u001b[0m After creating dir - [/var/hyperledger/production/ledgersData/chains/index/] exists"}
...

docker-compose-cli-logging.yaml

# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#

version: '2'

volumes:
  orderer.example.com:
  peer0.org1.example.com:
  peer1.org1.example.com:
  peer0.org2.example.com:
  peer1.org2.example.com:

networks:
  byfn:

services:

  orderer.example.com:
    extends:
      file:   base/docker-compose-base.yaml
      service: orderer.example.com
    container_name: orderer.example.com
    networks:
      - byfn
    depends_on:
      - logger
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  peer0.org1.example.com:
    container_name: peer0.org1.example.com
    extends:
      file:  base/docker-compose-base.yaml
      service: peer0.org1.example.com
    networks:
      - byfn
    depends_on:
      - logger
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  peer1.org1.example.com:
    container_name: peer1.org1.example.com
    extends:
      file:  base/docker-compose-base.yaml
      service: peer1.org1.example.com
    networks:
      - byfn
    depends_on:
      - logger
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  peer0.org2.example.com:
    container_name: peer0.org2.example.com
    extends:
      file:  base/docker-compose-base.yaml
      service: peer0.org2.example.com
    networks:
      - byfn
    depends_on:
      - logger
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  peer1.org2.example.com:
    container_name: peer1.org2.example.com
    extends:
      file:  base/docker-compose-base.yaml
      service: peer1.org2.example.com
    networks:
      - byfn
    depends_on:
      - logger
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  cli:
    container_name: cli
    image: hyperledger/fabric-tools:$IMAGE_TAG
    tty: true
    environment:
      - GOPATH=/opt/gopath
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_LOGGING_LEVEL=DEBUG
      - CORE_PEER_ID=cli
      - CORE_PEER_ADDRESS=peer0.org1.example.com:7051
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
      - CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
    command: /bin/bash -c './scripts/script.sh ${CHANNEL_NAME} ${DELAY}; sleep $TIMEOUT'
    volumes:
        - /var/run/:/host/var/run/
        - ./../chaincode/:/opt/gopath/src/github.com/hyperledger/fabric/examples/chaincode/go
        - ./crypto-config:/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/
        - ./scripts:/opt/gopath/src/github.com/hyperledger/fabric/peer/scripts/
        - ./channel-artifacts:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts
    depends_on:
      - logger
      - orderer.example.com
      - peer0.org1.example.com
      - peer1.org1.example.com
      - peer0.org2.example.com
      - peer1.org2.example.com
    networks:
      - byfn
    logging:
      driver: fluentd
      options:
        tag: docker.{{.ID}}

  logger:
    container_name: fluentd
    image: fluent/fluentd
    ports:
      - "24224:24224"
    volumes:
      - ./fluentd/log:/fluentd/log
      - ./fluentd/:/fluentd/etc
    environment:
      FLUENTD_CONF: fluentd.conf

fluentd.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match docker.**>
  @type stdout
</match>
#<match fluent.**>
#  @type stdout
#</match>