Notary guideline

notary-deployment

[TOC]

> additional important docs: [config references](https://github.com/docker/notary/tree/master/docs/reference)

## 0) Setting up DB(s)
1. Simply boot up two database instances (`i1` for server and `i2` for signer). 
2. create a database named `notaryserver` on `i1` and create `notarysigner` on `i2`. 
3. run SQL commands from [here](https://github.com/docker/notary/tree/master/migrations)
4. later configure the database urls in the `server-config.json` and `signer-config.json`
5. sit back and have a donut.

## 1) Setting up signer
1. `git clone https://github.com/cyc115/notary && git checkout cyc_prod_ready`
3. run `signer` with `docker-compose -f docker-compose.signer.yml up --build ` 
6. AWS instance need to expose inbound TCP `7789` for server-signer communication and outbound database port (default is `5432`)

### signer config file
```
{
  "server": {
    "grpc_addr": ":7899",
    "tls_cert_file": "./notary-signer.crt",
    "tls_key_file": "./notary-signer.key",
    "client_ca_file": "./notary-server.crt"
  },
  "logging": {
    "level": "debug"
  },
  "storage": {
    "backend": "postgres",
    "db_url": "postgres://user:pass@endpoint:port/notarysigner"
  }
}
```

## 2) Setting up server
1. `git clone https://github.com/cyc115/notary && git checkout cyc_prod_ready`
2. build the server with `docker-compose -f ./docker-compose.server.yml up --build`
3. Assign AWS in/out-bound rules to `signer` and database
4. ~~set up DNS in container to use `8.8.8.8` in `/etc/resolv.conf` to make sure signer is discoverable~~ (might not be necessary)
5. set up `notarysigner` in `/etc/hosts` to make sure the host name is known by server
6. create tls server key and cert pair using template below
7. test with notary cli client

### server config file (checkout up-to-date version in repo)
```
{
       "server": {
	            "http_addr": ":4443",
		        "tls_key_file" : "./cyc/key.pem",  
		        "tls_cert_file" : "./cyc/certificate.pem" 
		},
        "trust_service": {
                "type": "remote",
                "hostname": "notarysigner",
                "port": "7899",
                "tls_ca_file": "./root-ca.crt",
                "key_algorithm": "ecdsa",
                "tls_client_cert" :"./notary-server.crt",
                "tls_client_key": "./notary-server.key"
        },
        "logging": {
                "level": "info"
        },
        "storage": {
                "backend": "postgres",
                "db_url": "postgres://xx:xx@xxx/notaryserver"
        }
}
```
>  **note: ** in case of the following error: **`2017/06/19 22:37:13 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: lookup notarysigner on 127.0.0.11:53: no such host"; Reconnecting to {notarysigner:7899 <nil>}`. Set `8.8.8.8` as DNS either on host's `/etc/resolv.conf`, or in the corresponding docker-compose file.

## 3) set up daemon client
1. checkout `cyc_api_client` branch
3. config `Daemon server` TLS,  `escrow`in `clientapi-server-config.toml`
4. run `sudo docker-compose -f clientapi.yml up --build` to start `daemon` and `escrow`
6. AWS instance need to expose TCP inbound (4449) to allow cli connection

> - **NOTE:**  **Keys stored by `escrow` is stored in the mounted directory `.secrete`.  Key's encryption passphrase is stored as clear text in `clientapi-server-config.toml`. **
> - **IDEA** : implement a (reusable) configuration generation tool using vault as backend, this tool should take in a templated config file (json, yml, toml, etc), a secrete store service ( hasicorp vault, environment var, or a file from a mounted volume), smash them together and spit out ready to use configuration files. 
> - Another way is to use `docker-compose` ` [extend](https://blog.docker.com/2015/04/easily-configure-apps-for-multiple-environments-with-compose-1-2-and-much-more/) but it lacks flexibility. 


### clientapi-server.config 
```
[server]
grpc_addr = ":4449"
tls_cert_file = "./clientapi-server.crt"
tls_key_file = "./clientapi-server.key"

[logging]
level = "info"

[upstream]
addr = "https://54.174.67.41:4443"
tls_ca_file = "./clientapi-server.crt"
# addr = "notary-server:4443"

[key_storage]
type = "remote"
addr = "notaryescrow:4450"
tls_ca_file = "./root-ca.crt"
tls_cert_file = "./clientapi-server.crt.back"
tls_key_file = "./clientapi-server.key.back"
secret = "fixturesecret"
```

## 3) set up notary cli on local machine 
1. checkout `cyc_api_client`
2. run `make client`
3. make sure the local `~/.notary/config.json` file contains the correct daemon address. (Example shown below under `files` header) 
example CLI config: 
```
{
    "trust_dir" : "~/.docker/trust",
    "remote_server": {
        "url": "http://54.218.252.249:4443", 
        //tls stuff
    }
}
```

## 4) set up token based authentication

### 4.0) Authentication flow
```sequence
Daemon->NotaryServer: publish
NotaryServer->Daemon: 401 unauthorized
Daemon->AuthServer: authenticate
AuthServer->Daemon: Authorize + JWT
Daemon->NotaryServer: Publish w/ JWT
NotaryServer->NotaryServer: verify JWT w/ crt
NotaryServer->Daemon: 200 ok
```

### 4.1) Authorization server 

The Authorization server provides the following via `GET` request.

```javascript
{
	"token": "eyIn0.C1w.XXX",  // the only required field
	"access_token": "eyIn0.C1w.XXX",
	"expires_in": 300, // time to expire
	"issued_at": "2017-07-03T16:16:59.188578836Z" 
}
```
- **token** :  An opaque Bearer token that clients should supply to subsequent requests in the Authorization header.
- **access_token** : For compatibility with OAuth 2.0. When both **token** and **access_token** are specified, they should be equivalent; if they differ the client's choice is undefined.
- **expires_in**: (Optional) The duration in seconds since the token was issued that it will remain valid. When omitted, this defaults to 60 seconds. For compatibility with older clients, a token should never be returned with less than 60 seconds to live.
- **issued_at**:  (Optional) The RFC3339-serialized UTC standard time at which a given token was issued. If issued_at is omitted, the expiration is from when the token exchange completed.
- **refresh_token**:  (Optional) Token which can be used to get additional access tokens for the same subject with different scopes. This token should be kept secure by the client and only sent to the authorization server which issues bearer tokens. This field will only be set when `offline_token=true` is provided in the request.


### 4.2) JWT format 
the client would make GET request to the server with the following param:
```
https://auth.docker.io/token
    ?service=registry.docker.io
    &scope=repository:library/hello-world:pull,push`
```
the decrypted JWT token contains a `header`, `claim-set`, and `signature` :
#### header : 
```javascript
{
  "alg": "ES256",
  "typ": "JWT",
  "x5c": ["MIICL...yk="] // list of cert in trust chain
}
```

#### claim-set
```javascript
{
  "access": [{  //maybe empty if unauthorized
      "type": "repository",
      "name": "library/hello-world",
      "actions": [
        "pull"
      ]
    }], 
  "aud": "registry.docker.io",    //corresponds to the service in the GET param   
  "exp": 1497390356,
  "iat": 1497390056,
  "iss": "auth.docker.io",        // iss must match 
  "jti": "kWB3WGHvHxtO8GETn__x",
  "nbf": 1497389756,
  "sub": ""                       // sub must match
}
```

### 4.3) Notary Daemon setup 

Include the following in `clientapi-server-config.toml`

```toml
[auth]
type = "token"
    [auth.options]
    realm = "https://auth.docker.io/token" // where to fetch token
    service = "notary.docker.io" .         // request the audience field in the JWT, must match with server config
    issuer = "auth.docker.io"              // must match with server config
    rootcertbundle = "./dockerauth.crt"    // path relative to notary/fixtures
```


### 4.4) Notary Server setup

add the following to `server-config.json`
```javascript
"auth" : {
    "type": "token",
    "options": {
        "realm": "https://auth.docker.io",
        "service": "notary.docker.io",           // the audience field in the token
        "issuer": "auth.docker.io",
        "rootcertbundle": "./fixtures/xxx.crt" . // path relative to notary
    }
}
```

| Parameter | Required | Description                                           |
|-----------|----------|-------------------------------------------------------|
| `realm`   | yes      | The realm in which the registry server authenticates. |
| `service` | yes      | The service being authenticated.                      |
| `issuer`  | yes      | The name of the token issuer. The issuer inserts this into the token so it must match the value configured for the issuer. |
| `rootcertbundle` | yes | The absolute path to the root certificate bundle. This bundle contains the public part of the certificates used to sign authentication tokens. |

## files 

### `~/.notary/config.json`

```javascript
{
    "trust_dir": "~/.docker/trust",
    "remote_server": {
        "url": "https://54.174.67.41:4443",  //notary-server
        "root_ca": "./self-sign.pem"   // ca certificate used by conventional cli client to verify notary server 
    },
	// api client related 
    "api": {
        "addr" : "54.174.67.41:4449",
        "insecure" : false,
        "tls_ca_file": "/Users/xjqw46/.notary/self-sign.pem" 
    }
}
```
<table>
	<tr>
		<th>Parameter</th>
		<th>Required</th>
		<th>Description</th>
	</tr>
	<tr>
		<td valign="top"><code>url</code></td>
		<td valign="top">no</td>
		<td valign="top">URL of the Notary server: defaults to https://notary.docker.io
			This configuration option can be overridden with the command line flag
			`-s` or `--server`.</td>
	</tr>
	<tr>
		<td valign="top"><code>root_ca</code></td>
		<td valign="top">no</td>
		<td valign="top"><p>The path to the file containing the root CA with which to verify
			the TLS certificate of the Notary server, for example if it is self-signed.
			The path is relative to the directory of the configuration file.</p>
			<p>This configuration option can overridden with the command line flag
			`--tlscacert`, which would specify a path relative to the current working
			directory where the Notary client is invoked.</p>
			<p> <b>does not seem to work, require client to pass in rootcacert via flag </b></p>
	</td>
	</tr>
	<tr>
		<td valign="top"><code>tls_client_cert</code></td>
		<td valign="top">no</td>
		<td valign="top"><p>The path to the client certificate to use for mutual TLS with
			the Notary server.  Must be provided along with <code>tls_client_key</code>
			or not provided at all.  The path is relative to the directory of the
			configuration file.</p>
			<p>This configuration option can overridden with the command line flag
			`--tlscert`, which would specify a path relative to the current working
			directory where the Notary client is invoked.</p></td>
	</tr>
	<tr>
		<td valign="top"><code>tls_client_key</code></td>
		<td valign="top">no</td>
		<td valign="top"><p>The path to the client key to use for mutual TLS with
			the Notary server. Must be provided along with <code>tls_client_cert</code>
			or not provided at all.  The path is relative to the directory of the
			configuration file.</p>
			<p>This configuration option can overridden with the command line flag
			`--tlskey`, which would specify a path relative to the current working
			directory where the Notary client is invoked.</p></td>
	</tr>
</table>


### sample server cert config (client facing TLS cert) 
```
[ req ]
default_bits            = 4096
distinguished_name      = req_distinguished_name
req_extensions          = v3_req
x509_extensions         = v3_ca
prompt                  = no

[ v3_req ]
# required
subjectAltName = @alt_names 
basicConstraints = CA:TRUE
keyUsage = digitalSignature, keyEncipherment

[v3_ca]
# required
basicConstraints    = critical, CA:TRUE, pathlen:1
keyUsage            = critical, keyCertSign
subjectAltName      = @alt_names

[alt_names]
IP.1  = 54.174.67.41
DNS.1 = notaryserver
DNS.2 = notary-server
DNS.3 = notary-signer
DNS.4 = notarysigner

[ req_distinguished_name ]
0.organizationName        = Moto
organizationalUnitName    = unit name
emailAddress              = Yuehcuan.chen@motorolasolutions.com
localityName              = Chicago
stateOrProvinceName       = IL
countryName               = US
commonName                = 54.174.67.41
```

## example ca certificate used by server-signer verification
```
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8f:c7:59:4b:8d:49:44:23
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=CA, L=San Francisco, O=Docker, CN=Notary Testing CA
        Validity
            Not Before: Feb 17 00:43:13 2017 GMT
            Not After : Feb 15 00:43:13 2027 GMT
        Subject: C=US, ST=CA, L=San Francisco, O=Docker, CN=Notary Testing CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    7f:53:f4:3a:95:e1:e4:05:9c:fc:de:07:18:8c:09:
                    ....
                    7d:d3:4e:fb:95:9c:ed:2b:9a:c4:bb:03:dd:38:75:
                    07:9e:37
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Key Usage: critical
                Non Repudiation, Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                C5:DA:E5:75:FE:6D:AE:49:7F:46:11:B8:BE:94:AA:3B:54:41:2E:59
    Signature Algorithm: sha256WithRSAEncryption
         79:c9:b7:04:62:d6:8a:09:ac:34:87:8e:f8:f9:f4:d6:ff:84:
		 ....
         4b:02:26:91:11:c5:de:a1

```
## Other stuff
- Generating wildcard certificate
- Guideline for generating CA pin-able certificate
- setting up a health check 
- manage logs

![sit back and enjoy a donut](http://cdn.skim.gs/images/homer-simpson-doughnuts/what-homer-simpson-taught-us-about-doughnuts)

notary-deployment.md

[TOC]

Intro

server and signer will be deployed onto instance i1 and daemon escrow will be deployed onto i2.

Deploying with swarm mode provides 0 downtime capability and the potential to scale up to multiple instances when necessary. To use swarm however, we need to set up a private registry.

1) Setting up server and signer

0. make sure entry for notary-server is present in /etc/hosts
   1. make sure in ~/.notary/config.json remote_server$root_ca and remote_server$url are set to the server url and ca cert.
1. clone notary git clone https://github.com/cyc115/notary
2. checkout branchgit checkout cyc_prod_read. The branch has some minor changes to make sure server and signer delay connecting to DB.
3. configure server (server-config.json), signer (signer-config.json), and
4. build notary server/signer/db containers: docker-compose build, note: mysql is configured to allow empty password.
5. run server / signer and DB with sudo docker-compose start
6. AWS instance need to expose TCP inbound (4450)

Note: Moving notary Server from instance 2 to instance 1 is possible after server is finalized. We can then build an image out of server's docker file and then deploy server anywhere.

2) set up daemon client

1. checkout cyc_api_client branch
2. require clientapi.yml to be present (see files section )
3. config Daemon server TLS, escrowin clientapi-server-config.toml
4. run sudo docker-compose -f clientapi.yml up --build to start daemon and escrow
5. AWS instance need to expose TCP inbound (4449) and outbound port (4450)

• NOTE: keys are encrypted and stored in tmp/keys/dir in key escrow container. Keys are NOT PERSISTENT and are encrypted with a CONSTANT passphrase defined in clientapi-server-config.toml.
  • TODO: Clear text passphrase: write a script that runs on daemon boot that fetches the key and configuration from some secure place then execute the program.
    • Take advantage of swarm mode and "docker secrete"
  • DONE: Keys stored by escrow is stored in the mounted directory .secrete.

3) set up notary cli on local machine

1. checkout cyc_api_client
2. run make client
3. make sure the local ~/.notary/config.json file contains the correct daemon address. (Example shown below under files header)

XXX) set up token based authentication

add the following in clientapi-server-config.toml

[auth]
type = "token"
    [auth.options]   //JWT options
    realm = "https://auth.docker.io/token" // GET request to fetch token
    service = "notary.docker.io"
    issuer = "auth.docker.io"
    rootcertbundle = "./dockerauth.crt"

a GET request to the url specified by realm should return the following

{
	"token": "jwt token",
	"access_token": "same as above (optional)",
	"expires_in": 300, (optional) 
	"issued_at": "2017-06-13T21:42:48.903424078Z" (optional)
}

the decrypted JWT should have header, claim-set, and signature :

header :

{
  "alg": "ES256",
  "typ": "JWT",
  "x5c": ["MIICL...yk="] // list of cert in trust chain
}

claim-set

{
  "access": [], 
  "aud": "",
  "exp": 1497390356,
  "iat": 1497390056,
  "iss": "auth.docker.io",        // iss must match 
  "jti": "kWB3WGHvHxtO8GETn__x",
  "nbf": 1497389756,
  "sub": ""                       // sub must match
}

files

clientapi.yml on cyc_api_client branch

version: "2"
services:
  clientapi:
    build:
      context: .
      dockerfile: clientapi.Dockerfile
    ports:
      - "4449:4449"
    depends_on:
      - escrow
    networks:
      api:
      esc:
    entrypoint: /usr/bin/env sh
    command: -c "clientapi -config=./fixtures/clientapi-server-config.toml"
  escrow:
    build:
      context: .
      dockerfile: escrow.Dockerfile
    networks:
      esc:
        aliases:
          - notaryescrow
    entrypoint: /usr/bin/env sh
    command: -c "escrow -config=./fixtures/escrow-config-local.toml"
networks:
  api:
    external: false
  esc:
    external: false

~/.notary/config.json

{
    "trust_dir" : "~/.docker/trust",
    "remote_server": {
        "url": "https://notary-server:4443",
        "root_ca":  "/Users/xjqw46/go/src/github.com/docker/notary/fixtures/root-ca.crt"
    },
    "trust_pinning" : {
        "disable_tofu" : false
    },
    "api": {
        "addr" : "127.0.0.1:4449",
        "insecure" : false,
        "tls_ca_file": "/Users/xjqw46/go/src/github.com/docker/notary/fixtures/root-ca.crt"
    }
}

/etc/hosts

127.0.0.1 notary-server
127.0.0.1 notary-client

Other stuff

• Generating wildcard certificate
• Guideline for generating CA pin-able certificate
• setting up a health check
• load balancing and crash resistant : swarm?
• manage logs

VM splits

vm 1:

• server
• daemon
• server db
• key escrow
• persistant volume

vm2:

• signer db
• signer

example ca certificate

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8f:c7:59:4b:8d:49:44:23
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=CA, L=San Francisco, O=Docker, CN=Notary Testing CA
        Validity
            Not Before: Feb 17 00:43:13 2017 GMT
            Not After : Feb 15 00:43:13 2027 GMT
        Subject: C=US, ST=CA, L=San Francisco, O=Docker, CN=Notary Testing CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    7f:53:f4:3a:95:e1:e4:05:9c:fc:de:07:18:8c:09:
                    ....
                    7d:d3:4e:fb:95:9c:ed:2b:9a:c4:bb:03:dd:38:75:
                    07:9e:37
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Key Usage: critical
                Non Repudiation, Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                C5:DA:E5:75:FE:6D:AE:49:7F:46:11:B8:BE:94:AA:3B:54:41:2E:59
    Signature Algorithm: sha256WithRSAEncryption
         79:c9:b7:04:62:d6:8a:09:ac:34:87:8e:f8:f9:f4:d6:ff:84:
		 ....
         4b:02:26:91:11:c5:de:a1