168 lines
6.8 KiB
Markdown
168 lines
6.8 KiB
Markdown
# Package-Centric Source-Based Container Build System
|
|
|
|
## Motivation
|
|
* There's no standardized way to create container images that include
|
|
applications built from upstream sources.
|
|
* No standardized way to share artifacts for containers instead of whole
|
|
containers
|
|
* Application dependencies, which are mostly libraries, are typically
|
|
neglected when calculating container dependencies. This causes not knowing
|
|
what libraries are installed
|
|
|
|
## Features
|
|
### Package-based container builds
|
|
This project aims to find a solution for creating and managing containers in a package-centric manner.
|
|
In this context a package is both a definition of how to build (e.g. a Gentoo ebuild or Debian control file)
|
|
Every container will be able to specify their dependencies, which can be self-built packages or exact versions of already existing packages.
|
|
Like this the user will always exactly know what libraries, files, etc. is available to their container at runtime.
|
|
The specified container will have its dependencies on packages which themselves can be packaged separately into one container per package.
|
|
|
|
### Source-based builds
|
|
To allow maximum flexibility for users, the package files must be able to describe source builds, which allows the user to make changes to the source before the package is built and integrated into the target container image.
|
|
|
|
### Reproducibility
|
|
Builds of packages, as well as builds of container images, must be reproducible in a way that the same input always yields the same output.
|
|
Because every change of a package yields in a different identity, the system allows to have multiple versions of one software application.
|
|
These can differ in properties like source code patches, build configuration options, target architecture, etc..
|
|
|
|
### Portability of the whole system
|
|
As a portable system, builds can be processed locally on a user's workstation or distributed to a bunch of servers.
|
|
|
|
### Shareable source and binary files
|
|
The build system will ship tools that make it easy to discover and share source and binary files that are respectively consumed and produced by the build system.
|
|
This will allow for very fast setups of containers that involve only downloading from trusted repositories.
|
|
|
|
# Usage
|
|
## Buildit configuration
|
|
**.buildit-config.yaml**
|
|
```
|
|
---
|
|
repository:
|
|
name: mysuperbinhost
|
|
upload-type: ssh
|
|
upload-path: containers@mysuperbinhost.org/containers
|
|
download-type: https
|
|
download-path: mysuperbinhost.org/containers
|
|
```
|
|
|
|
## Sysadmin needs patched nginx
|
|
|
|
### Sysadmin
|
|
In case a sysadmin needs a patched and specifically configured version of its favorite webserver nginx.
|
|
|
|
1. Put directories and files in place
|
|
|
|
---
|
|
Directory layout
|
|
```
|
|
├── nginx-prod
|
|
│ ├── container.yaml
|
|
│ ├── files
|
|
│ │ └── nginx.conf
|
|
│ └── pkgs
|
|
│ └── nginx
|
|
│ ├── patches
|
|
│ │ └── https-only.patch
|
|
│ └── pkg.yaml
|
|
```
|
|
---
|
|
**pkg.yaml**
|
|
```
|
|
---
|
|
base: www-servers/nginx-1.7.6
|
|
author: Sysadmin42 <sys@admin42.org>
|
|
patches:
|
|
patches/https-only.patch: "This patch denies all plain http requests"
|
|
https://github.com/nginx/nginx/commit/52e4dc2f74fd032dace01acbe5eb29ddf7c1ad96.patch:
|
|
"Fix buffer overruns"
|
|
use:
|
|
with:
|
|
- ipv6
|
|
- selinux
|
|
|
|
```
|
|
---
|
|
**container.yaml**
|
|
```
|
|
---
|
|
- vars:
|
|
author: Sysadmin42
|
|
name: nginx-production
|
|
version: 1.7.6-p1
|
|
os: linux
|
|
arch: amd64
|
|
|
|
- package:
|
|
type: embedded
|
|
path: ./pkgs/nginx
|
|
|
|
- sync:
|
|
src: ./files/nginx.conf
|
|
dest: /etc/nginx/nginx.conf
|
|
recursive: True
|
|
chmod: 0644
|
|
|
|
- image:
|
|
type: aci
|
|
content: |
|
|
{
|
|
"acKind": "ImageManifest",
|
|
"acVersion": "0.6.1",
|
|
"name": "{{ name }}-{{ version }}",
|
|
"labels": [
|
|
{"name": "os", "value": "{{ os }}"},
|
|
{"name": "arch", "value": "{{ arch }}"}
|
|
],
|
|
"app": {
|
|
"exec": [
|
|
"/sbin/nginx"
|
|
],
|
|
"user": "0",
|
|
"group": "0"
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Build the container
|
|
```
|
|
$ buildit nginx-prod/ --discover=github.com/sysadmin42/containers,push=True
|
|
Building Sysadmin42/nginx-production-1.7.6-p1
|
|
Processing package from './pkgs/nginx' for linux/amd64.
|
|
HASH: 86c8ef43-f4a4-49ba-a0ee-92900211c7b6
|
|
Can't find HASH in any known location...
|
|
Defaulting to local build... [OK]
|
|
Uploading packages to 'mysuperbinhost' [OK]
|
|
Packaging Sysadmin42/nginx-production-1.7.6-p1 as ACI... [OK]
|
|
Uploading container spec and image(s) to 'mysuperbinhost' [OK]
|
|
```
|
|
|
|
# Implementation
|
|
## Resources
|
|
### CoreOS related tools
|
|
* https://github.com/derekchiang/acbuild
|
|
* https://github.com/coreos/manifest
|
|
|
|
### Similar Projects
|
|
* Previous work of mine: https://embedux.github.io/documentation/usage/rootfs/configuration.yml/index.html
|
|
|
|
### Operating Systems and Package managers
|
|
* http://nixos.org/nixos/about.html
|
|
* https://gitweb.gentoo.org/proj/releng.git/tree/releases/weekly/specs/amd64?id=HEAD
|
|
* https://github.com/zefhemel/nix-docker
|
|
* [nix build farm
|
|
paper](http://www.researchgate.net/publication/228629017_The_Nix_Build_Farm_A_declarative_approach_to_continuous_integration)
|
|
* https://blogs.gentoo.org/zmedico/2015/07/06/tardelta-generate-a-tarball-of-differences-between-two-tarballs/
|
|
* https://github.com/jordansissel/fpm/wiki
|
|
|
|
## Outlook
|
|
The completion of the described container build system will benefit greatly to how container images can be shared and deployed.
|
|
|
|
### Trusted Containers by reproducibility
|
|
Trusting container images has been hard. Being able to reproduce and verify the builds improves this.
|
|
|
|
### Obsolete Container-Vulnerabilities Scans
|
|
Vulnerabilities scans are only necessary if it's unknown what the container image contains. With the new build system the build specification allows to inspect the included container images much more efficiently. Image vendors can directly track contained packages and their CVEs instead of relying on posteriori scans.
|
|
|
|
### Automatic Container Updates
|
|
When identified, regular and security updates to 3rd party packages can trigger rebuilds as well as changed source files of 1st party applications. The update circle can be closed by automatically deploying new containers triggered by the updated images.
|
|
Complete automation might be difficult in real-world deployments because software updates sometimes require configuration changes.
|