class: center, middle, inverse ## How Red Hat builds containers .footnote[Vadim Rutkovsky
] .footnote[https://github.com/vrutkovs/slides-how-redhat-builds-containers] --- layout: false background-image: url(back.png) # Building an image is simple, right? * Take the dockerfile * And build it!: `docker build -t .` * Done! Oh, right, the image should have tags: `docker tag deadbeef myimage:latest` * Oops, the previous latest image was overwritten, I should add some more tags: `docker tag myimage:latest myimage:1.0` * Done, push to production instance! * Oh, wait, how do I start it? What version is it running anyway? #### Arrght, its too complicated, lets start from scratch! --- layout: false background-image: url(back.png) # What is wrong with simple Dockerfiles ```docker FROM centos:centos7 MAINTAINER The CentOS Project
RUN yum -y update; yum clean all RUN yum -y install epel-release; yum clean all RUN yum -y install mongodb-server; yum clean all RUN mkdir -p /data/db EXPOSE 27017 ENTRYPOINT ["/usr/bin/mongod"] ``` 1. Which version of mongo is installed? 1. Is it necessary to have a layer for each instruction? 1. Where do I find info about this package - URL, help file? 1. Why is `/data/db` being created there? Is it a mount point? 1. How do I start it with default config? with a custom config? ####Lets add the missing information --- background-image: url(back.png) # Add descriptive labels ```docker FROM centos:centos7 MAINTAINER The CentOS Project
RUN yum -y update && \ yum -y install epel-release \ mongodb-server && \ yum clean all && \ mkdir -p /data/db LABEL BZComponent="mongodb" LABEL Version="3.2.8" LABEL Release="1" LABEL HELP="Mongo (from 'humongous') is a high-performance, open source, schema-free document-oriented database. MongoDB is written in C++ and offers the following features: * Collection oriented storage: easy storage of object/JSON-style data * Dynamic queries * Full index support, including on inner objects and embedded arrays * Query profiling * Replication and fail-over support * Efficient storage of binary data including large objects (e.g. photos and videos) * Auto-sharding for cloud-level scalability (currently in early alpha) * Commercial Support Available A key goal of MongoDB is to bridge the gap between key/value stores (which are fast and highly scalable) and traditional RDBMS systems (which are deep in functionality)." LABEL RUN="docker run -ti -v /data:/data/db mongo-centos-7:latest" EXPOSE 27017 ENTRYPOINT ["/usr/bin/mongod"] ``` Now I should also update "Version" label whenever the package is updated --- background-image: url(back.png) # Building a decent image manually Build: ``` $ sudo docker build -t mongo-centos-7 . ``` Squash extra layers: ``` $ sudo docker history mongo-centos-7 . IMAGE CREATED CREATED BY SIZE 25954e6d2300 3 weeks ago /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/mongod 0 B 5ae69cb454a5 3 weeks ago /bin/sh -c #(nop) EXPOSE 27017 0 B ... b78336099045 3 weeks ago /bin/sh -c #(nop) MAINTAINER The CentOS Proje 0 B 4816a298548c 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B 6ee235cf4473 3 weeks ago /bin/sh -c #(nop) LABEL name=CentOS Base Imag 0 B 474c2ee77fa3 3 weeks ago /bin/sh -c #(nop) ADD file:72852fc7626d233343 196.6 MB 1544084fad81 6 months ago /bin/sh -c #(nop) MAINTAINER The CentOS Proje 0 B $ sudo docker-squash -f b78336099045 -t mongo-centos-7:squashed mongo-centos-7:latest $ sudo docker tag mongo-centos-7:squashed mongo-centos-7:latest ``` Tag with additional tags - for version, version-release and a random timestamp ```no-highlight $ sudo docker tag mongo-centos-7:latest mongo-centos-7:3.2.8 $ sudo docker tag mongo-centos-7:latest mongo-centos-7:3.2.8-1 $ sudo docker tag mongo-centos-7:latest mongo-centos-7:20170422171656 ``` Can this be automated somehow? --- class: center, middle background-image: url(back.png) ## Atomic Reactor .footnote[https://github.com/projectatomic/atomic-reactor] --- background-image: url(back.png) # What is Atomic Reactor Atomic Reactor is a python cli app which helps with building Docker images Features: 1. Builds images inside a container - the builds are separated between each other 1. Plugins - allows customizing the build process Installation: ``` $ pip install git+https://github.com/projectatomic/atomic-reactor.git@master ``` Prepare a buildroot - a docker image with necessary tools installed from a Dockerfile in 'images/dockerhost-builder': ``` $ sudo atomic-reactor create-build-image \ --reactor-local-path . \ ./images/dockerhost-builder \ buildroot ``` Build a Mongodb image using this buildroot: ``` $ sudo atomic-reactor build git \ --method hostdocker \ --build-image buildroot \ --image mongo-centos-7 \ --target-registries 172.17.42.1:5000 \ --uri "https://github.com/vrutkovs/mongodb-centos-7-docker.git" ``` --- background-image: url(back.png) # Build JSON Atomic reactor prepares a build json: ``` { "source": { "provider": "git", "provider_params": {"git_commit": "None" }, "uri": "https://github.com/vrutkovs/mongodb-centos-7-docker.git", "dockerfile_path": "None" }, "image": "mongo-centos-7", "prebuild_plugins": [ { "name": "pull_base_image", "args": { "parent_registry_insecure": "False", "parent_registry": "None" } } ], "postbuild_plugins": [ { "name": "tag_and_push", "args": { "registries": { "172.17.42.1:5000": { "insecure": "False" } } } } ] } ``` --- background-image: url(back.png) # What's happening inside atomic-reactor 1. Atomic reactor on the host starts a new container from buildroot image and passes the build json 1. Inside the container atomic reactor prepares a list of plugins to run from build json 1. The plugins are executed and the image is built Atomic reactor is able to run a build out of a custom JSON file too: ``` $ sudo atomic-reactor --verbose build json
--build-image buildroot --method hostdocker ``` --- background-image: url(back.png) # Most notable Atomic reactor plugins 1. Pre-build plugins: 1. **pull_base_image** - makes sure the base image is always fresh 1. **change_from_in_dockerfile** - update FROM line in dockerfile to container image ID 1. **add_labels_in_dockerfile** - add missing labels - build date, hostname, git reference, architecture 1. **add_dockerfile** - store the used dockerfile in the image 1. **add_help** - convert a specified markdown file in a man page (used in `atomic help` command) 1. **add_yum_repo_by_url** - install additional repo file and remove it in the end of the build 1. Post-build and pre-publish plugins 1. **squash** - layers created during the build are squashed into one 1. **tag_by_labels** - tag image based on Version and Release labels 1. **tag_and_push** - push the image it to the specified repository or repositories 1. **all_rpm_packages** - gather info about RPM packages 1. Exit plugins 1. **koji_promote** - create a build in Koji buildsystem with a tarball of the image and required metadata 1. **sendmail** - send an email about the status of the build --- background-image: url(back.png) class: center, middle ## OSBS .footnote[OpenShift Build System] --- background-image: url(back.png) # OSBS details #### Build environment Openshift - can be easily extended, has monitoring and metrics. Openshift-specific objects like BuildConfig and ImageStream are used in OSBS #### A tool which would create build jsons from a template and start builds [OSBS-client](https://github.com/projectatomic/osbs-client) creates a build json based on parameters, has API to be used in other projects #### Integration with existing build system [Koji](https://fedoraproject.org/wiki/Koji) is used in Red Hat / Fedora / CentOS to build RPMs, ISOs, Maven projects and can be extended to build container images --- background-image: url(back.png) # Developer workflow 1. Developer commits a Dockerfile, pushes it and runs ``` $ fedpkg container-build ``` 1. Local plugin is verifying that koji build target (based on the branch name) is valid and creates a `buildContainer` task on the koji 1. [Koji-containerbuild plugin](https://github.com/release-engineering/koji-containerbuild) checks that the image has necessary labels and uses OSBS-client API to start a new build: 1. OSBS-Client generates a new build json based on the received parameters and internal configuration 1. OSBS-Client creates a BuildConfig on Openshift with necessary parameters 1. OSBS-client starts a new build on Openshift cluster and sends logs from the build to the build system 1. Openshift picks the less busy node, starts a new container from the buildroot image 1. The container starts Atomic Reactor, builds the image and runs specified plugins 1. After the build is complete Koji task is finished, a new koji build is created 1. Koji build URL and pullspecs are shown in the CLI --- background-image: url(back.png) # Orchestration Orchestration workflow allows us to use other Openshift clusters as workers Enables load balancing, post-build assembly operations on orchestrator cluster and multiple availability zones for workers. #### Example: Multi-arch image building: 1. Orchestrator cluster receives a request for multi-arch build 1. Verifies the source code, fetches packages 1. Picks workers for required architecture - worker_x86_64 and worker_aarch64 1. Workers build images in parallel, push arch-specific images to the registry 1. Orchestrator waits until all workers are finished 1. Orchestrator prepares the multi-arch manifest (manifest list) - a manifest which points to arch-specific image depending on which arch is used on the container engine requesting this image. 1. Orchestrator tags the manifest list, creates Koji build etc. --- background-image: url(back.png) # Chained rebuilds, isolated builds OSBS leverages Openshift ImageStreams to trigger builds whenever base image is updated: 1. When a child image is built a new imagestream pointing to a parent build is created 1. Whenever a new parent image version is released, a new Openshift build can be triggered 1. Atomic Reactor verifies that this package has autorebuild enabled - the build goes on if it is 1. After a successful build a new build in Koji is created, as if it was manually triggered ##### Isolated builds Some tasks, however, require a shortcut version of this workflow - for instance, for quick CVE build. A special - isolated - build can be created on request. This isolated build would repeat the build for given image or koji buikd, but with different repo, pointing to updated packages. This allows us to release CVE fixes faster. --- background-image: url(back.png) class: center, middle ## Demo time! --- background-image: url(back.png) # Future features #### Image signing and container scanning Build phase can be extended to include vulnerability scanning and image signing #### OCI images Docker build is implemented as a plugin in Atomic Reactor, so other types of buildstep plugins are possible - to build ansible-container, OCI images (e.g. Flatpaks) --- background-image: url(back.png) class: center, middle ## That's all folks