How to create a FreeBSD application build farm with Poudriere
Managing software is an essential part of any organisation. Building your own software ensures you have more control over the features and versions your users have access to and the timing of any upgrades. This article offers insights into the benefits of managing your own software with practical examples so you can create and manage your own BSD build farm.
After reading this article you will:
- Understand the advantages of build farms and be able to asses your requirements;
- Understand the different virtualisations offered by BSD jails and Docker images;
- Understand, configure and use pkg to install application;
- Be able to create a Poudriere build farm on FreeBSD;
- Be able to securely sign packages;
- Be able to make packages available for distribution;
- Be able to upgrade ports;
- Be able to manage a list of ports to build;
- Be able to monitor build progress;
- Be able to monitor installed ports.
Advantages of build farms and why you might want to build your own apps
There are many options and choices when building software. Do you want a general app that can do everything? Perhaps you prefer a small footprint? Do you require and older or newer version of Python? Does your estate use Open LDAP/Active Directory for single sign-on/account authentication? These and many more question might be good reasons to build your own apps.
Before package managers, BSD apps came as "ports" ready to be built and installed by calling "make install". For most ports, a pre-compiled package also exists, saving the work and time of having to compile anything at all. FreeBSD offer both the latest available packages and a more conservative quarterly branch with default options. Sometimes you need something different - This is when you start building your own.
Poudriere came into existence to fulfil this need, it's both flexible and powerful. You can find it in /usr/ports/ports-mgmt/poudriere.
Choosing your preferred versions and upgrading apps at your own pace. Poudriere offers the ability to build different sets of apps so you could, for example, build with a default python, 3.9 as of this article, and also an older version 3.8 so you can manage a mixed estate with a segment of your existing apps using a previous version as well as another segment using the latest.
Maintaining obsolete or custom apps that are not part of the FreeBSD ports collection. Many apps can be maintained and built using current compilers and take advantage of shared libraries maintained as part of the ports collection.
FreeBSD Poudriere & Jails Vs Linux Docker
Poudriere builds packages for use by real or virtual machines, Docker builds apps in self contained images that can be managed by tools such as Kubernetes, Swarm, Nomad, and ECS.
The end result is similar, how they are used and how to get there is different.
Poudriere builds an app then packages it up for distribution. A package contains a list of requirements so it's easy to get an app up and running by simply calling pkg install [name], configuring it then launching the application with service [name] start. It can be run either in a real *BSD machine, a lightweight jail or virtual machine. Orchestrating this process is straightforward, my preference is Salt Project but there are many excellent choices available.
The end result is a list of packages that can be used to build and run many lightweight jails in a variety of configurations.
There is a clear separation between apps built with poudriere and the BSD jails they run on, with full audit trail build logs available for inspection.
Docker is built up of layers, build stages can be used to compile an app using a development environment, then copy the application over with just a runtime reducing excessive disc space.
Docker builds a unique image that can be used to launch many instances. Docker compose can create and launch multi container images.
Introducing the BSD package system
pkg was released in 2012 and the modern FreeBSD package system as we know it today was born.
It's a sophisticated package management system, it can be used as a command line utility, it's part of many configuration management apps and built into most graphical desktop environments including Gnome and KDE. The majority of its facilities are outside the scope of this article.
In essence, it enables creating and maintaining a list of available packages that can be installed, and makes installing and maintaining packages on a system or VM easy. FreeBSD packages are available from local mirrors https://docs.freebsd.org/en/books/handbook/mirrors/.
For more information on pkg see https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8&format=html
Development is on GitHub https://github.com/freebsd/pkg
Listing versions that need upgrading
Discovering what ports are installed and if they need updating
# pkg version --verbose --ports --not-like =
akonadi-22.04.2_1 < needs updating (port has 22.04.3_2) cmake-3.23.2 < needs updating (port has 3.23.3)
This listing shows there are two ports that can be upgraded. The following would start the upgrade process, you will be presented with a list of packages that will be upgraded, answering Y will cause the upgrades to proceed.
# pkg upgrade
It's possible to build a port on a machine by changing into a port then typing make install. In that case you would use pkg version --index to use the local ports tree.
The --not-like = supresses a list of current sofware and just shows two ports need updating.
Bootstrapping the package system
https://docs.freebsd.org/en/books/handbook/ports/#pkgng-intro
To bootstrap the system, run:
# /usr/sbin/pkg
You'll find the default FreeBSD package configuration file at:
/usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}
Updating ports
Updating ports is an easy command line option:
# pkg update
If you're using salt, it's simply the following:
# salt --async '*' pkg.upgrade
Auditing software against vulnerabilities
All software has bug, but some are more serious than others.
To check installed software against known vulnerabilities:
# pkg audit
git-2.36.1_1 is vulnerable:
git -- privilege escalation
CVE: CVE-2022-29187
WWW: https://vuxml.FreeBSD.org/freebsd/b99f99f6-021e-11ed-8c6f-000c29ffbb6c.html
This report is included in the default daily security run. You can include or exclude reports by editing /etc/periodic.conf adding any of the following:
- daily_backup_pkg_jails="YES"
- daily_backup_pkg_enable="YES"
- daily_status_pkg_changes_enable="YES"
- security_status_pkgaudit_enable="YES"
- security_status_baseaudit_enable="YES"
- security_status_pkg_checksum_enable="YES"
- weekly_status_pkg_enable="YES"
Create a FreeBSD build farm using Poudriere
Poudriere is a bulk package builder and port tester. Originally designed to test package production, it has full compilation audit logs. The majority of its features are outside the scope of this article, but if you're interested in development or investigating issues, then I highly recommend exploring it further https://www.freshports.org/ports-mgmt/poudriere. This article will cover installing, configuring, creating a jail then building a list of ports.
Bootstrapping poudriere by either calling:
# pkg install poudriere
Or by changing into /usr/ports/ports-mgmt/poudriere and calling
# make; make install
Once installed, the first thing to do is edit the configuration file, the sample is a good starting point.
/usr/local/etc/poudriere.conf.sample
/usr/local/etc/poudriere.conf
It's important to refer to the poudriere man page and also the man pages referred to in SEE ALSO. Many options can be configured for each set, jail and ports tree.
Here are a few notable configuration options:
If you're using zfs, you must set ZPOOL
ZPOOL=zroot
You must tell fetch where to find your closest mirror. You can find a list of mirrors and the protocols they offer here https://docs.freebsd.org/en/books/handbook/mirrors/. For example here in the UK the entry is:
FREEBSD_HOST=ftp://ftp.uk.freebsd.org
Poudriere can be optimised, but if you are running on a machine with limited memory, perhaps less than 32Gb memory, then you may find changing USE_TMPFS or switching it off may allow some resource intensive ports to build.
USE_TMPFS=no
If you have a slow machine
# This defines the max time (in seconds) that a command may run for a build
# before it is killed for taking too long. Default: 86400
#MAX_EXECUTION_TIME=86400
MAX_EXECUTION_TIME=345600
# This defines the time (in seconds) before a command is considered to
# be in a runaway state for having no output on stdout. Default: 7200
#NOHANG_TIME=7200
NOHANG_TIME=14400
Also where your signing key can be found. This is covered in "Securely sign packages"
PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key
Create a default ports tree
First create a default ports tree
# poudriere ports -c -p default
You can see the newly created ports tree:
# poudriere ports -l
default git+https 2022-08-16 15:57:25 /usr/local/poudriere/ports/default
Make a list of ports to build
Create a list of ports you would like to make available. The following is a simple list to demonstrate how dependencies are managed and options are configured.
/usr/local/etc/poudriere-default.packages
databases/postgresql14-contrib
databases/postgresql14-docs
databases/postgresql14-pgtcl
databases/postgresql14-server
sysutils/py-salt
www/py-django40
www/py-djangorestframework
Note that the postgresql client is not included.
Configuring all ports
If you wish to set software versions when building ports you can set them in /usr/local/etc/poudriere.d/make.conf, for example:
WITH_COLLATION=utf8_general_ci
CUPS_OVERWRITE_BASE=yes
DEFAULT_VERSIONS+= ssl=base
DEFAULT_VERSIONS+= apache=2.4
DEFAULT_VERSIONS+= mysql=5.7
DEFAULT_VERSIONS+= pgsql=14
They can be set for jails, ports trees, and sets. Look for "Create optional make.conf" in man poudriere.
Configuring individual ports
To configure the Django port:
# poudriere options -n -c www/py-django40
You will be presented with the following choices
poudriere-django-options.png
Select DOCS, HTMLDOCS, PGSQL and unselect SQLITE.
Create the first build jail
This example is FreeBSD 13.1 amd64
# poudriere jail -c -j FreeBSD:13:amd64 -v 13.1-RELEASE
[00:00:00] Creating FreeBSD:13:amd64 fs at /usr/local/poudriere/jails/FreeBSD_13_amd64... done
[00:00:00] FREEBSD_HOST from config invalid; defaulting to https://download.FreeBSD.org
[00:00:00] Using pre-distributed MANIFEST for FreeBSD 13.1-RELEASE amd64
[00:00:00] Fetching base for FreeBSD 13.1-RELEASE amd64
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr 186 MB 4545 kBps 41s
[00:00:44] Extracting base... done
[00:01:19] Fetching src for FreeBSD 13.1-RELEASE amd64
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr 183 MB 4570 kBps 42s
[00:02:02] Extracting src... done
[00:03:45] Fetching lib32 for FreeBSD 13.1-RELEASE amd64
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr 63 MB 4617 kBps 14s
[00:04:00] Extracting lib32... done
[00:04:14] Cleaning up... done
[00:04:14] Recording filesystem state for clean... done
[00:04:14] Upgrading using http
/etc/resolv.conf -> /usr/local/poudriere/jails/FreeBSD_13_amd64/etc/resolv.conf
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 13.1-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 18 patches.....10.... done.
Applying patches... done.
The following files will be updated as part of updating to
13.1-RELEASE-p1:
/bin/freebsd-version
/usr/lib/lib9p.a
/usr/lib/lib9p.so.1
/usr/lib/lib9p_p.a
/usr/lib/libpam.a
/usr/lib/pam_exec.so.6
/usr/lib32/lib9p.a
/usr/lib32/lib9p.so.1
/usr/lib32/lib9p_p.a
/usr/lib32/libpam.a
/usr/lib32/pam_exec.so.6
/usr/src/contrib/lib9p/pack.c
/usr/src/lib/libpam/modules/pam_exec/pam_exec.c
/usr/src/sys/cam/cam_periph.c
/usr/src/sys/conf/newvers.sh
/usr/src/sys/kern/imgact_elf.c
/usr/src/sys/kern/kern_event.c
/usr/src/sys/vm/vm_fault.c
Installing updates...Scanning //usr/share/certs/blacklisted for certificates...
Scanning //usr/share/certs/trusted for certificates...
done.
13.1-RELEASE-p1
[00:04:22] Recording filesystem state for clean... done
[00:04:22] Jail FreeBSD:13:amd64 13.1-RELEASE-p1 amd64 is ready to be used
The jail is created and ready to start building
# poudriere jail -l
FreeBSD:13:amd64 13.1-RELEASE-p1 amd64 http 2022-08-16 16:22:31 /usr/local/poudriere/jails/FreeBSD_13_amd64
A useful reminder that url_base should be set in poudriere.conf to where you would like to view build progress:
URL_BASE=https://poudriere.example.com/
A few more things to set-up before the build is started.
Securely sign packages
Packages can be signed using public/provate keys as identified in poudriere.conf:
PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key
Generate a private key using openssl
openssl genrsa -out myrepo.key 2048
Ensure it's owned by root and has read only permission
mv myrepo.key /etc/ssl/keys/
chmod 0400 /etc/ssl/keys/myrepo.key
Let poudriere.conf know where to find the private key
PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key
Generate the public key
openssl rsa -in myrepo.key -out myrepo.pub -pubout
Copy the public key to /etc/ssl/keys/:
cp myrepo.key /etc/ssl/keys/
Ensure it's readable by anybody
chmod 0444 /etc/ssl/keys/myrepo.key
Then let pkg (/usr/local/etc/pkg/repos/myrepo.conf) know where to find it:
pubkey: "/etc/ssl/keys/myrepo.pub",
Distribute your public key
There is a chicken and egg situation, in order to install a port you need to check the package with your public key before installation, but you need your packages installed to distribute your public key.
No doubt you have your own bootstrapping process when installing machines and creating jails/VMs. Distributing package public keys is included in infrastructure management to ensure consistency and facilitate change. Use your preferred infrastructure management / configuration management tool. Here are the instructions for Salt project:
Copy the public key for public distribution, it's treated like other public certificates and in this configuration certs.sls is included in top.sls.
cp myrepo.pub /usr/local/etc/salt/states/certs/
certs.sls
/etc/ssl/keys/myrepo.pub:
file.managed:
- source: salt://certs/myrepo.pub
- makedirs: True
Package management
Package configuration files live in /usr/local/etc/pkg/repos/ and are named like the following:
/usr/local/etc/pkg/repos/myrepo.conf
The /usr/local/etc/pkg/repos/FreeBSD.conf is shown above, a local myrepo.conf might look like the following:
myrepo: {
url: "https://packages.example.com/${ABI}-default",
mirror_type: "none",
enabled: yes,
signature_type: "pubkey",
pubkey: "/etc/ssl/keys/myrepo.pub",
debug_level: 4,
}
The ABI is derived from /usr/bin/uname and looks like the following: FreeBSD:14:amd64.
After you have build your packages and are ready to switch from packages.freebsd.org to your own, ensure yours is set to enabled and edit /usr/local/etc/pkg/repos/FreeBSD.conf to "enabled: NO":
FreeBSD: { enabled: NO }
Start a package build
Start building your default packages with the bulk command:
poudriere bulk -j FreeBSD:14:amd64 -f /usr/local/etc/poudriere-default.packages
This will create the reference jail and start building the listed packages.
Making package build progress available
Any web server that can host static files can surface poudriere build progress.
Using apache as an example:
<VirtualHost *:443>
ServerName packages.example.com
ServerAdmin admin@example.com
DocumentRoot /usr/local/poudriere/data/packages/
SSLCertificateFile /path/to/example.com.crt
SSLCertificateKeyFile /path/to/example.com.key
SSLEngine on
<Directory "/usr/local/poudriere/data/packages">
Options None +FollowSymLinks
SetHandler default-handler
Require all granted
AllowOverride None
</Directory>
</VirtualHost>
Pointing your browser to your https://packages.example.com should present you with the latest builds for your jail and any build statistics.
Making packages available
After packages have been built, they need to be accessible across your estate.
If you only have one machine, or want to retrieve packages locally, the url can be a file location, for example your myrepo.conf might be:
url: "file:///usr/local/poudriere/data/packages/FreeBSD:13:amd64-default",
Most of the time you might want to use a generic one using the ABI:
url: "https://packages.example.com/${ABI}-default",
Any web server can be used to distribute packages, here is a simple Apache one:
<VirtualHost *:443>
ServerName packages.example.com
ServerAdmin admin@example.com
DocumentRoot /usr/local/poudriere/data/packages/
SSLCertificateFile /path/to/example.com.crt
SSLCertificateKeyFile /path/to/example.com.key
SSLEngine on
<Directory "/usr/local/poudriere/data/packages">
Options None +FollowSymLinks
SetHandler default-handler
Require all granted
AllowOverride None
</Directory>
</VirtualHost>
If your poudriere jails are not named according to the ABI "FreeBSD:14:amd64" then simply add aliases as necessary
Alias /FreeBSD:13:amd64 /usr/local/poudriere/data/packages/13amd64-default
Alias /FreeBSD:13:arm64 /usr/local/poudriere/data/packages/13arm64-default
You might wish to proxy requests for different architectures and protect by ip address using something like the following Apache virtual host.
<VirtualHost *:443>
ServerName packages.example.com
ServerAdmin admin@example.com
UseCanonicalName On
DocumentRoot /usr/local/www/example.com/public_html
SSLCertificateFile /path/to/example.com.crt
SSLCertificateKeyFile /path/to/example.com.key
SSLEngine on
SSLProxyEngine on
SSLProxyCheckPeerName off
<Location "/FreeBSD:13:amd64">
<RequireAny>
Require ip 2001:470:1f1d:362::5/128
Require ip 192.168.1.0/24
</RequireAny>
ProxyPass "https://jail-amd64.example.com/"
ProxyPassReverse "https://jail-amd64.example.com/"
</Location>
<Location "/FreeBSD:13:arm64">
<RequireAny>
Require ip 2001:470:1f1d:362::5/128
Require ip 192.168.1.0/24
</RequireAny>
ProxyPass "https://jail-arm64.example.com/"
ProxyPassReverse "https://jail-arm64.example.com/"
</Location>
</VirtualHost>
Keeping up to date
It's essential to keep two ports trees. The first is the default ports tree at /usr/ports, the second is your default poudriere ports tree at /usr/local/poudriere/ports/default.
The first you keep up to date, perhaps weekly, so you know what is currently available. It's essential to read /usr/ports/UPDATING.
The second is your default poudriere ports tree that you will only update when you are ready to start a new build run. Remember building ports is under your control.
Upgrading a local ports tree in /usr/ports
If you don't have a working ports tree, then clone a new one
git clone https://git.FreeBSD.org/ports.git /usr/ports
Then update as necessary:
git pull --ff-only
You can check to see which ports can be upgraded
pkg version -v --index
xrandr-1.5.1 = up-to-date with index
xwayland-devel-21.0.99.1.211 < needs updating (index has 21.0.99.1.262)
When you are ready to upgrade the following will upgrade your default poudriere ports tree:
poudriere ports -u
Check any information in /ports/UPDATING then run:
poudriere bulk -j [jail name] -f [your.packages]
Checking progress via your website https://poudriere.example.com/
When complete, you can upgrade using your normal configuration management, or your users can upgrade as normal with either a GUI or command line:
pkg upgrade
Summary
You now have a stable BSD build farm managing your applications. It's auditable and easy to manage.
Apps are continually changing and evolving. As they are now under your control, you'll find your own cadence for updating, building and distributing them.