Skip to content

Instantly share code, notes, and snippets.

@maxamillion

maxamillion/osbs_notes_v2.md

Last active November 2, 2016 21:28
Show Gist options
  • Save maxamillion/d990c7a6b0ae035fd6a6 to your computer and use it in GitHub Desktop.
Save maxamillion/d990c7a6b0ae035fd6a6 to your computer and use it in GitHub Desktop.
Download ZIP
osbs_notes_v2.md
Raw
osbs_notes_v2.md

#Notes for OSBS with Docker Registry V2

Pre-Reqs

Build atomic-reactor dev

## If you have not done this already, clone the git repository
## (need my fork for Fedora build fixes until merged upstream)
$ git clone https://github.com/maxamillion/atomic-reactor.git
$ git checkout build

## Build SRPM for use with COPR
$ tito build --srpm --test

Submit a build to COPR

Build osbs-client dev

## If you have not done this already, clone the git repository
## (need my clone for the tito init and Fedora build fixes until merged upstream)
$ git clone https://github.com/maxamillion/osbs-client.git
$ git checkout build

## Build SRPM for use with COPR
$ tito build --srpm --test

Submit a build to COPR

Setting up a test Fedora hosted OSBS instance for local Fedora builds

Install Fedora 23+ with hostname osbs.localdomain

$ ip addr
$ vi /etc/hosts  # add 'osbs.localdomain' for this IP address
$ dnf update -y --refresh
$ systemctl reboot

$ systemctl enable sshd.service
$ systemctl start sshd.service
$ mkdir .ssh; chmod 700 .ssh
$ ssh-keygen # If needed
$ scp .ssh/id_rsa.pub root@osbs.localdomain:.ssh/authorized_keys #why not ssh-copy-id?;)

## Setup dev build COPR
$  dnf copr enable maxamillion/atomic-reactor 

## Setup openshift install source for openshift (Fedora 23 only, Fedora 24/Rawhide has it in official repos)
$ dnf copr enable maxamillion/fedora-openshift 

## bootstrap ansible for F23+
$ dnf install -y ansible python2-dnf libselinux-python libsemanage-python python-firewall

OpenShift with OSBS configuration setup

# OPENSHIFT
# https://github.com/projectatomic/ansible-osbs

$ git clone https://github.com/projectatomic/ansible-osbs.git
$ cd ansible-osbs
./update-roles.sh
$ vim hosts  # change 'rhel71' to 'osbs.localdomain' on line 2
$ vim group_vars/all
  # at line 9: 'atomic_reactor_source: git'
  # at line 17, uncomment section
$ ansible-playbook -i hosts site.yml

# Re-source ~/.bashrc to pick up the KUBECONFIG env var that ansible playbooks place
$ . ~/.bashrc

Setup atomic-reactor buildroot with dev build from COPR

$ dnf -y install atomic-reactor
$ atomic-reactor create-build-image --reactor-tarball-path /usr/share/atomic-reactor/atomic-reactor.tar.gz /usr/share/atomic-reactor/images/dockerhost-builder buildroot

Docker v2 Registry Setup

Option 1 - pulp/pulp-crane

For V2-only support, the two steps are:

  1. Install Pulp using the right versions:

    • pulp 2.6
    • pulp-docker from branch docker_v2_api
    • python-crane 1.2.0

    and also install pulp-docker-admin-extensions for 'pulp-admin docker ...' in order to poke at it

  2. In your osbs.conf instance definition, set:

registry_api_versions = v2


Note that you will still need to run a docker-distribution instance in
order to transfer V2 content, as the only way to get the content into
Pulp is to have it sync from a V2 registry.

The client atomic-reactor uses to talk to Pulp is dockpulp, so you'll
also need that, https://github.com/release-engineering/dockpulp.

You want to build an /etc/dockpulp.conf into the buildroot too:

[pulps] pulp-registry = https://pulp/ distribution = https://distribution/

[registries] pulp-registry = https://pulp/v1/repositories # not sure if this should be v2 distribution = https://distribution/

[filers] pulp-registry = https://pulp/ distribution = https://distribution/

[verify] pulp-registry = no distribution = no


Adjust to taste. Here, pulp is the pulp server and distribution is
running docker-distribution. pulp-registry is an arbitrary name for the
configuration.

You need to create a Kubernetes secret containing the Pulp certificate
and key. To do that:

dock-pulp.py -s pulp login ... oc secrets new pulp-secret pulp.cer=/.pulp/pulp.cer pulp.key=/.pulp/pulp.key oc secrets add serviceaccount/builder secrets/pulp-secret --for=mount


pulp-secret is an arbitrary name for the secret.

Then you can set these `osbs.conf` keys for your instance:

pulp_registry_name = pulp-registry pulp_secret = pulp-secret



### Option 2 - docker-distribution

DOCKER DISTRIBUTION

https://github.com/docker/distribution/blob/master/docs/insecure.md#user-content-using-self-signed-certificates

$ cd ~/ # this should be /root/ because you should be root to run this $ mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt # use osbs.localdomain for Common Name

$ mkdir -p /etc/docker/certs.d/osbs.localdomain:5000 $ cp certs/domain.crt /etc/docker/certs.d/osbs.localdomain:5000/ca.crt $ trust anchor certs/domain.crt

$ docker run -d -p 5000:5000 --restart=always --name registry -v /root/certs:/certs:z -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2

fedora into v2 registry

$ docker pull fedora $ docker tag fedora osbs.localdomain:5000/fedora $ docker push osbs.localdomain:5000/fedora


## OpenShift image stream setup

create 'fedora' image stream

cat <<"EOF" | oc create -f - { "apiVersion": "v1", "kind": "ImageStream", "metadata": { "name": "fedora" }, "spec": { "dockerImageRepository": "osbs.localdomain:5000/fedora" } } EOF


## OSBS Client setup

Implement authentication with client certificates.

To test:

* Create cert/key pair for your OpenShift instance for user user_name
  and place them in the api_config directory:


  oadm create-api-client-config \
    --user=user_name \
    --client-dir=api_config \
    --certificate-authority=/etc/openshift/master/ca.crt \
    --signer-cert=/etc/openshift/master/ca.crt \
    --signer-key=/etc/openshift/master/ca.key \
    --signer-serial=/etc/openshift/master/ca.serial.txt


* Copy the api_config directory to your machine.
* Use osbs-client with client certificate authentication:


  osbs \
    -i yourinstance \
    --client-cert api_config/user_name.crt \
    --client-key api_config/user_name.key \
    --without-auth \
    get-user


  You should see Name: "user_name" in the output. Note that OAuth is not
  used here.

$ dnf install osbs-client -y

Note: The following builder_openshift_url = https://172.17.42.1:8443/ is for Docker < 1.9.1

For Docker >= 1.9.1 it needs to be builder_openshift_url = https://172.17.0.1:8443/

$ cat <<"EOF" > /etc/osbs.conf [general] verbose=1 build_json_dir = /usr/share/osbs/ openshift_required_version = 1.0.8

[default] openshift_url = https://osbs.localdomain:8443/ kubelet_uri = http://osbs.localdomain:10250/ (not needed anymore) koji_root = http://koji.fedoraproject.org/koji koji_hub = http://koji.fedoraproject.org/kojihub sources_command = fedpkg sources build_type = prod registry_uri = https://osbs.localdomain:5000/v2 source_registry_uri = https://osbs.localdomain:5000/v2 authoritative_registry = registry.example.com vendor = Fedora Project build_host = osbs.localdomain verify_ssl = false use_auth = false builder_use_auth = true distribution_scope = private registry_api_versions = v2 builder_openshift_url = https://172.17.0.1:8443/ EOF


## Kick Off a Demo Build

Kick off a demo build

$ osbs --conf /etc/osbs.conf build --git-url https://github.com/twaugh/docker-hello-world --git-branch master --user admiller --component docker-hello-world --target rawhide



## Known Issues with workrounds

### port 8443 refused from inside the build container

2015-12-14 23:19:51,857 - atomic_reactor.plugin - DEBUG - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugin.py", line 203, in run plugin_response = plugin_instance.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugins/exit_store_metadata_in_o sv3.py", line 148, in run osbs.set_annotations_on_build(build_id, labels, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 28, in catch_exceptions return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 315, in set_annotations_on_build return self.os.set_annotations_on_build(build_id, annotations, namespace=namespace) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 328, in set_annotations_on_build build_json = self._get(url).json() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 85, in _get headers, kwargs = self._request_args(with_auth, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 73, in _request_args self.get_oauth_token() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 104, in get_oauth_token r = self._get(url, with_auth=False, allow_redirects=False) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 86, in _get return self._con.get(url, headers=headers, verify_ssl=self.verify_ssl, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/http.py", line 412, in get return self._do_request(url, "get", **kwargs) File "/usr/lib/python2.7/site-packages/osbs/http.py", line 403, in _do_request raise OsbsNetworkException(url, message, code, *ex.args[2:]) OsbsNetworkException: (7) Failed to connect to osbs.localdomain port 8443: Connection refused


Solution: Make sure to set the following line in `/etc/osbs.conf` under `[default]`

builder_openshift_url = https://172.17.42.1:8443/


If using Docker >= 1.10 make sure this is:

builder_openshift_url = https://172.17.0.1:8443/


### KeyError('access_token',) from atomic_reactor

2015-12-14 23:22:11,877 - atomic_reactor.plugins.store_metadata_in_osv3 - INFO - build id = docker-hello-world-master-2 2015-12-14 23:22:11,878 - atomic_reactor.core - INFO - inspecting image 'fedora:latest' 2015-12-14 23:22:11,879 - atomic_reactor.core - DEBUG - image_id = 'fedora:latest' 2015-12-14 23:22:11,935 - atomic_reactor.plugin - DEBUG - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugin.py", line 203, in run plugin_response = plugin_instance.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugins/exit_store_metadata_in_o sv3.py", line 148, in run osbs.set_annotations_on_build(build_id, labels, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 39, in catch_exceptions raise OsbsException(cause=ex, traceback=sys.exc_info()[2]) OsbsException: KeyError('access_token',)

Original traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osbs/api.py", line 28, in catch_exceptions return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 315, in set_annotations_on_build return self.os.set_annotations_on_build(build_id, annotations, namespace=namespace) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 328, in set_annotations_on_build build_json = self._get(url).json() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 85, in _get headers, kwargs = self._request_args(with_auth, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 73, in _request_args self.get_oauth_token() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 118, in get_oauth_token self.token = parsed_fragment[b'access_token'][0] KeyError('access_token',)


Solution: make sure to set the following in `/etc/osbs.conf` under `[default]` section:

builder_use_auth = false


## Known Issues without currently known workarounds

### OsbsNetworkException: (404) and atomic_reactor.plugin.PluginFailedException: plugin 'squash' raised an exception

Reported Upstream:
https://github.com/projectatomic/osbs-client/issues/331

2016-01-04 20:59:19,725 - atomic_reactor.plugin - DEBUG - running plugin 'store_metadata_in_osv3' 2016-01-04 20:59:19,727 - atomic_reactor.plugin - INFO - running plugin instance with args: '{u'url': u'https://172.17.42.1:8443/', u'verify_ssl': False, u'use_auth': True}' 2016-01-04 20:59:19,729 - atomic_reactor.plugins.store_metadata_in_osv3 - INFO - build id = docker-hello-world-master-1 2016-01-04 20:59:52,808 - atomic_reactor.plugin - DEBUG - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugin.py", line 203, in run plugin_response = plugin_instance.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugins/exit_store_metadata_in_osv3.py", line 148, in run osbs.set_annotations_on_build(build_id, labels, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 28, in catch_exceptions return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/api.py", line 315, in set_annotations_on_build return self.os.set_annotations_on_build(build_id, annotations, namespace=namespace) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 328, in set_annotations_on_build build_json = self._get(url).json() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 85, in _get headers, kwargs = self._request_args(with_auth, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 73, in _request_args self.get_oauth_token() File "/usr/lib/python2.7/site-packages/osbs/core.py", line 104, in get_oauth_token r = self._get(url, with_auth=False, allow_redirects=False) File "/usr/lib/python2.7/site-packages/osbs/core.py", line 86, in _get return self._con.get(url, headers=headers, verify_ssl=self.verify_ssl, **kwargs) File "/usr/lib/python2.7/site-packages/osbs/http.py", line 412, in get return self._do_request(url, "get", **kwargs) File "/usr/lib/python2.7/site-packages/osbs/http.py", line 403, in _do_request raise OsbsNetworkException(url, message, code, *ex.args[2:]) OsbsNetworkException: (7) Failed to connect to 172.17.42.1 port 8443: No route to host

2016-01-04 20:59:52,809 - atomic_reactor.plugin - WARNING - plugin 'store_metadata_in_osv3' raised an exception: OsbsNetworkException(u'(7) Failed to connect to 172.17.42.1 port 8443: No route to host',) 2016-01-04 20:59:52,809 - atomic_reactor.plugin - INFO - error is not fatal, continuing... 2016-01-04 20:59:52,809 - atomic_reactor.plugin - DEBUG - running plugin 'remove_built_image' 2016-01-04 20:59:52,811 - atomic_reactor.plugin - INFO - running plugin instance with args: '{}' 2016-01-04 20:59:52,811 - atomic_reactor.core - INFO - removing image 'sha256:99117e1c60a0687509ca4a9e042b7a9963e4b91f971894cd5fc117795c0e2eef' from filesystem 2016-01-04 20:59:52,811 - atomic_reactor.core - DEBUG - image_id = 'sha256:99117e1c60a0687509ca4a9e042b7a9963e4b91f971894cd5fc117795c0e2eef' 2016-01-04 21:01:24,800 - atomic_reactor.core - INFO - removing image 'fedora:latest' from filesystem 2016-01-04 21:01:24,800 - atomic_reactor.core - DEBUG - image_id = 'fedora:latest' 2016-01-04 21:01:24,812 - atomic_reactor.core - INFO - removing image 'osbs.localdomain:5000/fedora:latest' from filesystem 2016-01-04 21:01:24,812 - atomic_reactor.core - DEBUG - image_id = 'osbs.localdomain:5000/fedora:latest' Traceback (most recent call last): File "/usr/bin/atomic-reactor", line 9, in load_entry_point('atomic-reactor==1.6.0', 'console_scripts', 'atomic-reactor')() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 299, in run cli.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 282, in run args.func(args) File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 89, in cli_inside_build build_inside(input_method=args.input, input_args=args.input_arg, substitutions=args.substitute) File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/inner.py", line 402, in build_inside build_result = dbw.build_docker_image() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/inner.py", line 344, in build_docker_image prepublish_runner.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugin.py", line 222, in run raise PluginFailedException(msg) atomic_reactor.plugin.PluginFailedException: plugin 'squash' raised an exception: IOError(2, 'No such file or directory') 2016-01-04 21:01:24,975 - osbs.http.curl - DEBUG - b'Closing connection 0' 2016-01-04 21:01:24,976 - osbs.http - DEBUG - end of the stream 2016-01-04 21:01:24,976 - osbs.http - DEBUG - cleaning up


### atomic_reactor RuntimeError("Missing label 'name'.",)

Traceback (most recent call last): File "/usr/bin/atomic-reactor", line 9, in load_entry_point('atomic-reactor==1.6.0', 'console_scripts', 'atomic-reactor')() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 299, in run cli.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 282, in run args.func(args) File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/cli/main.py", line 89, in cli_inside_build build_inside(input_method=args.input, input_args=args.input_arg, substitutions=args.substitute) File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/inner.py", line 402, in build_inside build_result = dbw.build_docker_image() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/inner.py", line 352, in build_docker_image postbuild_runner.run() File "/usr/lib/python2.7/site-packages/atomic_reactor-1.6.0-py2.7.egg/atomic_reactor/plugin.py", line 222, in run raise PluginFailedException(msg) atomic_reactor.plugin.PluginFailedException: plugin 'tag_by_labels' raised an exception: RuntimeError("Missing label 'name'.",)


### Unable to connect to server to handle "imagestreams"

cat <<"EOF" | oc create -f -

{ "apiVersion": "v1", "kind": "ImageStream", "metadata": { "name": "fedora" }, "spec": { "dockerImageRepository": "osbs.localdomain:5000/fedora" } } EOF unable to connect to a server to handle "imagestreams": No configuration file found, please login or point to an existing file:

  1. Via the command-line flag --config
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.


## Deploy koji-containerbuild plugin

https://github.com/release-engineering/koji-containerbuild

Install `koji-containerbuild` from my [COPR](https://copr.fedoraproject.org/coprs/maxamillion/atomic-reactor/)

$ yum -y install koji-containerbuild koji-containerbuild-{builder,cli,hub}

All hub machines need koji-containerbuild-hub and all builders that will be

enabled for building containers need koji-containerbuild-builder

	
Enabled the plugins for hub and builder (kojid) on respective hosts

In `/etc/kojid/kojid.conf` add `builder_containerbuild` to the plugins line:

plugins = builder_container

        

In `/etc/koji-hub/hub.conf` add `hub_containerbuild` to the plugins line:

plugins = hub_containerbuild


On for the client stuff you'll need to add the following to `/etc/koji.conf` (changing the needed bits)

[koji-containerbuild]

;configuration for koji cli tool

;url of XMLRPC server server = http://devops.example.com/kojihub

;url of web interface weburl = http://devops.example.com/koji

;url of package download site topurl = https://kojipkgs.localdomain/

;path to the koji top directory ;topdir = /mnt/koji

;configuration for Kerberos authentication

;the service name of the principal being used by the hub ;krbservice = host

;configuration for SSL authentication

;client certificate cert = ~/.koji/client.crt

;certificate of the CA that issued the client certificate ca = ~/.koji/serverca.crt

;certificate of the CA that issued the HTTP server certificate serverca = ~/.koji/serverca.crt


### Setup Koji tag, target, docker "packages" - Option 1
As the kojiadmin, add builder(s) to the channel and add a package

This setup is for inheriting from f24 and all it's content. This is ideal but we're waiting on signed repos to be implemented that make it work properly.

The --new here will create the channel if it doesn't exist already

$ koji add-host-to-channel kojibuilder1.stg.fedoraproject.org container --new

$ stg-koji add-tag f24-docker --parent f24

Note: The arches field below should reflect the currently supported arch list for

$ stg-koji add-tag f24-docker-build --parent f24-build --arches x86_64

$ stg-koji add-target f24-docker-candidate f24-docker-build f24-docker

$ stg-koji add-pkg --owner maxamillion f24-docker $PKG_NAME

$ stg-koji grant-cg-access containerbuild atomic-reactor --new


### Setup Koji tag, target, docker "packages" - Option 2
As the kojiadmin, add builder(s) to the channel and add a package

This setup does not inherit from f24. This is to work around the fact that we don't have koji signed repos and will instead inject alternate repos into the atomic-reactor buildroot

The --new here will create the channel if it doesn't exist already

$ koji add-host-to-channel kojibuilder1.stg.fedoraproject.org container --new

$ stg-koji add-tag f24-docker

Note: The arches field below should reflect the currently supported arch list for

$ stg-koji add-tag f24-docker-build --arches x86_64 --parent=f24-docker

$ stg-koji add-target f24-docker-candidate f24-docker-build f24-docker

$ stg-koji add-pkg --owner maxamillion f24-docker $PKG_NAME

$ stg-koji grant-cg-access containerbuild atomic-reactor --new

$ stg-koji add-external-repo -t f24-docker kojipkgs-everything-f24-branched 'https://kojipkgs.fedoraproject.org/pub/fedora/linux/development/24/Everything/$arch/os/'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment