As part of the OCP Security Project the Platform Services (PS) team recognized that many of the images in the openshift namespace are not actively maintained. This is problematic because they are used by other teams leading to the propagation of security vulnerabilities; this will cause widespread security issues that will not be manageable when Aqua is put in play.
Too greatly improve our security posture while not making our customers lives overly complicated or negatively impacting the platforms usability will require thinking about image management differently. Rather than one vast dumping ground for images, curated or not, we’ll use the following:
The sample images shipped in the openshift namespace are a convenient way to spin up an image for testing or for actual production use and because of teams’ history with OCP3.11; many deployment manifests will be configured to pull images from this namespace. Going forward, the openshift namespace will be highly curated. This will help disuade teams for implementing bad practices and stream line images (seporate the wheat form the chaff).
- The
openshiftnamespace will start empty, and contain no images; - Once we determine that an image should be added to this namespace it will be manually added;
- Once manually added it will be automatically maintained by OCP so that updates are automatically imported.
The strategy above will be maintained on all clusters.
When teams “run a build” the output is an image, stored in an image stream, that can be run on the cluster. Typically teams will store images in their tools namespace and tag them with dev, test, or prod to trigger a deployment as per a BuildConfig. While there are variations on this strategy, the main take away is that teams will be encouraged to store images on the cluster and within their namespaces for use.
When Artifactory becomes available we require all images run in a *-prod to originate from Artifactory because a centrelized image repos is esential for multi-cluster deployments.
The workflow will be:
- Teams will build images in their
*-toolsnamespace; - Teams will manage how images are promoted (copy to dev, test, prod or just refernce tools);
- Without Artifactory, teams will need to propogate these image to other clusters.
- With Artifactory, teams will need to propogate images to Artifactory for production deployemnts.
Open Questions:
- Can we configure other clusters to import teams images every n-minutes?
Platform services will create a bcgov repository on all clusters where images curated by the PS team will be stored. These will include, for example, any builder images or common services like the container backup images or our preferred configuration for Patroni. The images will be built in a single locaiton (lab cluster) and propocated to the same image repository on all other clusters.
There will be a few conventions for all images in this namespace including:
- All images need to have a primary PS contact / owner;
- All images must be periodically rebuilt as part of a scheduled CI/CD pipeline;
- Images follow a semantic versioning guideline.
Community contributions will be welcome here providing they meet the same guidelines and are actively supported. It will be important for community contributions to have a PS champion because community team members come and go. The PS team member will need to be able to rebuild community around abandon images or make the decision to deprecate the image and remove the image.
Aqua will be used to scan images and enforce security policies set by the PS team in consultaiton with other CITZ security teams. It will be too resource intensive for Aqua to scan all images periodically. To manage security and resource usage the following process will be implemented:
- Teams will be given the ability to scan images produced as part of their build;
- Aqua will scan all images run in
*-prodnamespaces and block new images with excessive security vulnerabilities from being run.
2020-09-11
- Need to figure out how to integrate CICD with Aqua; no Aqua on Day 1.
- Image versioning sample:
patroni-postgres-96:v1.1; We will, in general, follow RHs name and version format (ie https://catalog.redhat.com/software/containers/rhel8/postgresql-96/5ba0ad1f5a134643ef2eeb9d). - Jeff / Cailey: Req. meeting to hammer out life custom image life cycel:
-
- How long will we keep a version around? Clecio suggests max 3.
-
- How do we enguage the community to take over image management if they want additional features such as update to patronie. We'll maintain CICD pipeline and distribution.
- Should not advertize that we have
imagestreamtagsync enabled. We may require teams to use Artifactory for pulling in external images. - Need to look into how supporting images are used for builds (ie S2I) so the custer can build if the external images are unavailable. (ref. Image: registry.redhat.io/openshift3/ose-docker-builder:v3.11.232).
- Images in
openshiftnamespace requried for the cluster: cli cli-artifacts installer installer-artifacts must-gather oauth-proxy tests tools - Do we need to do anything for Jenkins to work?
- We would like to optimize templates so that they play nice on the cluster.
- Remove jenkins from catalog and make ours avail from
bcgovwhich may or may not be changed; the template will be for sure. - Our existing pipeline: https://jenkins-bcgov-tools.pathfinder.gov.bc.ca/
- Justin to check with Aqua to see if we can use something like a regex to specify namespaces to scan like
*-prod; - Jason to develop semantic versioning guideline for PS images;
- Jason to develop image management guidelines for PS team members;
- Steven to create
bcgovnamespace (same rback so all can access read) on all clusters; - Justin to help write ticket on how to disemenate images from klab tools to all clusters;
- Jason to creat a ticket for what curated images we will keep; PS team to sugest images for the
openshiftrepo (list below); - Justin / Steven to check how we can keep OCP required images but not make them avail to all teams;
- Jason to build CICD pipeline klab for the minio image (look at our bcgov tools repo already);
- Patrick / Shelly to socialize image management plan for OCP4 via OCP 101 training classes;
- Steven to verify sample images auto update; - DONE.
- Cailey / Justin operator and gitops mechanics to provision an artifactory access credential for each newly provisioned namespace;
- TBD Investiage pointing Artifactory at the cluster for OIDC to avoid having to provision a SA to each team;
- Justin / RH / Steven to implement mechanics so we can build images with licenced entitlements;
- TBD create a better deployment tempalte for the Jenkens image with more sane resource limits
oc -n openshift get template | grep jenkins. - Jason test the OCP4 Jenkins image to see if it works resonably well for teams that on-board; we would like to be ?ephemeral?. Check with Patrick. Make sure template has sane resource usage (update or replace template with our own).
- Jason how do we seporate templates (repo) from the images in
openshfit?
PS Curated Images or Template (ordered by priority):
- Jenkens (maybe customize) - Jason
- Container backup-image(s) (good 1st for community support) - Wade maybe
- Patroni - Jason Low Priorotiy:
- Caddy S2I (v1, v2)
- Minio
- NATS
- nginx
| Remove | Name |
|---|---|
| [x] | apicast-gateway |
| [x] | apicurito-ui |
| [x] | cli |
| [x] | cli-artifacts |
| [ ] | dotnet |
| [ ] | dotnet-runtime |
| [x] | eap-cd-openshift |
| [x] | eap-cd-runtime-openshift |
| [x] | fis-java-openshift |
| [x] | fis-karaf-openshift |
| [x] | fuse-apicurito-generator |
| [x] | fuse7-console |
| [x] | fuse7-eap-openshift |
| [x] | fuse7-java-openshift |
| [x] | fuse7-karaf-openshift |
| [ ] | golang |
| [x] | httpd |
| [x] | installer |
| [x] | installer-artifacts |
| [ ] | java |
| [ ] | jboss-amq-62 |
| [ ] | jboss-amq-63 |
| [x] | jboss-datagrid65-client-openshift |
| [x] | jboss-datagrid65-openshift |
| [x] | jboss-datagrid71-client-openshift |
| [x] | jboss-datagrid71-openshift |
| [x] | jboss-datagrid72-openshift |
| [x] | jboss-datagrid73-openshift |
| [x] | jboss-datavirt64-driver-openshift |
| [x] | jboss-datavirt64-openshift |
| [x] | jboss-decisionserver64-openshift |
| [ ] | jboss-eap64-openshift |
| [ ] | jboss-eap70-openshift |
| [ ] | jboss-eap71-openshift |
| [ ] | jboss-eap72-openshift |
| [x] | jboss-fuse70-console |
| [x] | jboss-fuse70-eap-openshift |
| [x] | jboss-fuse70-java-openshift |
| [x] | jboss-fuse70-karaf-openshift |
| [x] | jboss-processserver64-openshift |
| [ ] | jboss-webserver30-tomcat7-openshift |
| [ ] | jboss-webserver30-tomcat8-openshift |
| [ ] | jboss-webserver31-tomcat7-openshift |
| [ ] | jboss-webserver31-tomcat8-openshift |
| [ ] | jboss-webserver50-tomcat9-openshift |
| [ ] | jenkins |
| [ ] | jenkins-agent-maven |
| [ ] | jenkins-agent-nodejs |
| [x] | mariadb |
| [x] | modern-webapp |
| [ ] | mongodb |
| [x] | must-gather |
| [ ] | mysql |
| [ ] | nginx |
| [ ] | nodejs |
| [ ] | oauth-proxy |
| [ ] | openjdk-11-rhel7 |
| [ ] | openjdk-11-rhel8 |
| [ ] | openjdk-8-rhel8 |
| [x] | perl |
| [x] | php |
| [ ] | postgresql |
| [ ] | python |
| [ ] | redhat-openjdk18-openshift |
| [x] | redhat-sso70-openshift |
| [x] | redhat-sso71-openshift |
| [x] | redhat-sso72-openshift |
| [x] | redhat-sso73-openshift |
| [ ] | redis |
| [x] | rhdm-decisioncentral-rhel8 |
| [x] | rhdm-kieserver-rhel8 |
| [x] | rhdm-optaweb-employee-rostering-rhel8 |
| [x] | rhpam-businesscentral-monitoring-rhel8 |
| [x] | rhpam-businesscentral-rhel8 |
| [x] | rhpam-kieserver-rhel8 |
| [x] | rhpam-smartrouter-rhel8 |
| [ ] | ruby |
| [ ] | sso74-openshift-rhel8 |
| [x] | tests |
| [x] | tools |
As of writing Artifactory is not available and will require issues to be fixed and an HA instance to be created. When this happens the above mentiond plan will be updated.
These are the images not managed by the sample operator, and must stay
Also, can you add a table of the Templates that are managed by the samples operator and which should be removed?