How to create a FreeBSD application build farm with Poudriere

Managing software is an essential part of any organisation. Building your own software ensures you have more control over the features and versions your users have access to and the timing of any upgrades. This article offers insights into the benefits of managing your own software with practical examples so you can create and manage your own BSD build farm.

FreeBSD Build Farm: Beastie watering various apps including Python, PostgreSQL, Apache, Qt and wxWidgets

After reading this article you will:

  • Understand the advantages of build farms and be able to asses your requirements;
  • Understand the different virtualisations offered by BSD jails and Docker images;
  • Understand, configure and use pkg to install application;
  • Be able to create a Poudriere build farm on FreeBSD;
  • Be able to securely sign packages;
  • Be able to make packages available for distribution;
  • Be able to upgrade ports;
  • Be able to manage a list of ports to build;
  • Be able to monitor build progress;
  • Be able to monitor installed ports.

Advantages of build farms and why you might want to build your own apps

There are many options and choices when building software. Do you want a general app that can do everything? Perhaps you prefer a small footprint? Do you require and older or newer version of Python? Does your estate use Open LDAP/Active Directory for single sign-on/account authentication? These and many more question might be good reasons to build your own apps.

Before package managers, BSD apps came as "ports" ready to be built and installed by calling "make install". For most ports, a pre-compiled package also exists, saving the work and time of having to compile anything at all. FreeBSD offer both the latest available packages and a more conservative quarterly branch with default options. Sometimes you need something different - This is when you start building your own.

Poudriere came into existence to fulfil this need, it's both flexible and powerful. You can find it in /usr/ports/ports-mgmt/poudriere.

Choosing your preferred versions and upgrading apps at your own pace. Poudriere offers the ability to build different sets of apps so you could, for example, build with a default python, 3.9 as of this article, and also an older version 3.8 so you can manage a mixed estate with a segment of your existing apps using a previous version as well as another segment using the latest.

Maintaining obsolete or custom apps that are not part of the FreeBSD ports collection. Many apps can be maintained and built using current compilers and take advantage of shared libraries maintained as part of the ports collection.

FreeBSD Poudriere & Jails Vs Linux Docker

Poudriere builds packages for use by real or virtual machines, Docker builds apps in self contained images that can be managed by tools such as Kubernetes, Swarm, Nomad, and ECS.

The end result is similar, how they are used and how to get there is different.

Poudriere builds an app then packages it up for distribution. A package contains a list of requirements so it's easy to get an app up and running by simply calling pkg install [name], configuring it then launching the application with service [name] start. It can be run either in a real *BSD machine, a lightweight jail or virtual machine. Orchestrating this process is straightforward, my preference is Salt Project but there are many excellent choices available.

The end result is a list of packages that can be used to build and run many lightweight jails in a variety of configurations.

There is a clear separation between apps built with poudriere and the BSD jails they run on, with full audit trail build logs available for inspection.

Docker is built up of layers, build stages can be used to compile an app using a development environment, then copy the application over with just a runtime reducing excessive disc space.

Docker builds a unique image that can be used to launch many instances. Docker compose can create and launch multi container images.

Introducing the BSD package system

pkg was released in 2012 and the modern FreeBSD package system as we know it today was born.

It's a sophisticated package management system, it can be used as a command line utility, it's part of many configuration management apps and built into most graphical desktop environments including Gnome and KDE. The majority of its facilities are outside the scope of this article.

In essence, it enables creating and maintaining a list of available packages that can be installed, and makes installing and maintaining packages on a system or VM easy. FreeBSD packages are available from local mirrors https://docs.freebsd.org/en/books/handbook/mirrors/.

For more information on pkg see https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8&format=html

Development is on GitHub https://github.com/freebsd/pkg

Listing versions that need upgrading

Discovering what ports are installed and if they need updating

# pkg version --verbose --ports --not-like =
akonadi-22.04.2_1                  <   needs updating (port has 22.04.3_2)
cmake-3.23.2                       <   needs updating (port has 3.23.3)

This listing shows there are two ports that can be upgraded. The following would start the upgrade process, you will be presented with a list of packages that will be upgraded, answering Y will cause the upgrades to proceed.

# pkg upgrade

It's possible to build a port on a machine by changing into a port then typing make install. In that case you would use pkg version --index to use the local ports tree.

The --not-like = supresses a list of current sofware and just shows two ports need updating.

Bootstrapping the package system

https://docs.freebsd.org/en/books/handbook/ports/#pkgng-intro

To bootstrap the system, run:
# /usr/sbin/pkg

You'll find the default FreeBSD package configuration file at:

/usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

Updating ports

Updating ports is an easy command line option:
# pkg update

If you're using salt, it's simply the following:
# salt --async '*' pkg.upgrade

Auditing software against vulnerabilities

All software has bug, but some are more serious than others.

To check installed software against known vulnerabilities:

# pkg audit
git-2.36.1_1 is vulnerable:
  git -- privilege escalation
  CVE: CVE-2022-29187
  WWW: https://vuxml.FreeBSD.org/freebsd/b99f99f6-021e-11ed-8c6f-000c29ffbb6c.html

This report is included in the default daily security run. You can include or exclude reports by editing /etc/periodic.conf adding any of the following:

  • daily_backup_pkg_jails="YES"
  • daily_backup_pkg_enable="YES"
  • daily_status_pkg_changes_enable="YES"
  • security_status_pkgaudit_enable="YES"
  • security_status_baseaudit_enable="YES"
  • security_status_pkg_checksum_enable="YES"
  • weekly_status_pkg_enable="YES"

Create a FreeBSD build farm using Poudriere

Poudriere is a bulk package builder and port tester. Originally designed to test package production, it has full compilation audit logs. The majority of its features are outside the scope of this article, but if you're interested in development or investigating issues, then I highly recommend exploring it further https://www.freshports.org/ports-mgmt/poudriere. This article will cover installing, configuring, creating a jail then building a list of ports.

Bootstrapping poudriere by either calling:

# pkg install poudriere

Or by changing into /usr/ports/ports-mgmt/poudriere and calling

# make; make install

Once installed, the first thing to do is edit the configuration file, the sample is a good starting point.

/usr/local/etc/poudriere.conf.sample
/usr/local/etc/poudriere.conf

It's important to refer to the poudriere man page and also the man pages referred to in SEE ALSO. Many options can be configured for each set, jail and ports tree.

Here are a few notable configuration options:
If you're using zfs, you must set ZPOOL

ZPOOL=zroot

You must tell fetch where to find your closest mirror. You can find a list of mirrors and the protocols they offer here https://docs.freebsd.org/en/books/handbook/mirrors/. For example here in the UK the entry is:

FREEBSD_HOST=ftp://ftp.uk.freebsd.org

Poudriere can be optimised, but if you are running on a machine with limited memory, perhaps less than 32Gb memory, then you may find changing USE_TMPFS or switching it off may allow some resource intensive ports to build.

USE_TMPFS=no

If you have a slow machine
# This defines the max time (in seconds) that a command may run for a build
# before it is killed for taking too long. Default: 86400

#MAX_EXECUTION_TIME=86400
MAX_EXECUTION_TIME=345600

# This defines the time (in seconds) before a command is considered to
# be in a runaway state for having no output on stdout. Default: 7200

#NOHANG_TIME=7200
NOHANG_TIME=14400

Also where your signing key can be found. This is covered in "Securely sign packages"

PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key

Create a default ports tree

First create a default ports tree

# poudriere ports -c -p default

You can see the newly created ports tree:

# poudriere ports -l 
default   git+https 2022-08-16 15:57:25 /usr/local/poudriere/ports/default

Make a list of ports to build

Create a list of ports you would like to make available. The following is a simple list to demonstrate how dependencies are managed and options are configured.

/usr/local/etc/poudriere-default.packages
databases/postgresql14-contrib
databases/postgresql14-docs
databases/postgresql14-pgtcl
databases/postgresql14-server
sysutils/py-salt
www/py-django40
www/py-djangorestframework

Note that the postgresql client is not included.

Configuring all ports

If you wish to set software versions when building ports you can set them in /usr/local/etc/poudriere.d/make.conf, for example:

WITH_COLLATION=utf8_general_ci 
CUPS_OVERWRITE_BASE=yes 
DEFAULT_VERSIONS+= ssl=base 
DEFAULT_VERSIONS+= apache=2.4 
DEFAULT_VERSIONS+= mysql=5.7 
DEFAULT_VERSIONS+= pgsql=14

They can be set for jails, ports trees, and sets. Look for "Create optional make.conf" in man poudriere.

Configuring individual ports

To configure the Django port:

# poudriere options -n -c www/py-django40

You will be presented with the following choices

poudriere-django-options.png

Select DOCS, HTMLDOCS, PGSQL and unselect SQLITE.

Create the first build jail

This example is FreeBSD 13.1 amd64

# poudriere jail -c -j FreeBSD:13:amd64 -v 13.1-RELEASE  
[00:00:00] Creating FreeBSD:13:amd64 fs at /usr/local/poudriere/jails/FreeBSD_13_amd64... done 
[00:00:00] FREEBSD_HOST from config invalid; defaulting to https://download.FreeBSD.org 
[00:00:00] Using pre-distributed MANIFEST for FreeBSD 13.1-RELEASE amd64 
[00:00:00] Fetching base for FreeBSD 13.1-RELEASE amd64 
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr         186 MB 4545 kBps    41s 
[00:00:44] Extracting base... done 
[00:01:19] Fetching src for FreeBSD 13.1-RELEASE amd64 
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr         183 MB 4570 kBps    42s 
[00:02:02] Extracting src... done 
[00:03:45] Fetching lib32 for FreeBSD 13.1-RELEASE amd64 
/usr/local/poudriere/jails/FreeBSD_13_amd64/fr          63 MB 4617 kBps    14s 
[00:04:00] Extracting lib32... done 
[00:04:14] Cleaning up... done 
[00:04:14] Recording filesystem state for clean... done 
[00:04:14] Upgrading using http 
/etc/resolv.conf -> /usr/local/poudriere/jails/FreeBSD_13_amd64/etc/resolv.conf 
Looking up update.FreeBSD.org mirrors... 2 mirrors found. 
Fetching public key from update2.freebsd.org... done. 
Fetching metadata signature for 13.1-RELEASE from update2.freebsd.org... done. 
Fetching metadata index... done. 
Fetching 2 metadata files... done. 
Inspecting system... done. 
Preparing to download files... done. 
Fetching 18 patches.....10.... done. 
Applying patches... done. 
The following files will be updated as part of updating to 
13.1-RELEASE-p1:
/bin/freebsd-version 
/usr/lib/lib9p.a 
/usr/lib/lib9p.so.1 
/usr/lib/lib9p_p.a 
/usr/lib/libpam.a 
/usr/lib/pam_exec.so.6 
/usr/lib32/lib9p.a 
/usr/lib32/lib9p.so.1 
/usr/lib32/lib9p_p.a 
/usr/lib32/libpam.a 
/usr/lib32/pam_exec.so.6 
/usr/src/contrib/lib9p/pack.c 
/usr/src/lib/libpam/modules/pam_exec/pam_exec.c 
/usr/src/sys/cam/cam_periph.c 
/usr/src/sys/conf/newvers.sh 
/usr/src/sys/kern/imgact_elf.c 
/usr/src/sys/kern/kern_event.c 
/usr/src/sys/vm/vm_fault.c 
Installing updates...Scanning //usr/share/certs/blacklisted for certificates... 
Scanning //usr/share/certs/trusted for certificates... 
done. 
13.1-RELEASE-p1 
[00:04:22] Recording filesystem state for clean... done 
[00:04:22] Jail FreeBSD:13:amd64 13.1-RELEASE-p1 amd64 is ready to be used

The jail is created and ready to start building

# poudriere jail -l
FreeBSD:13:amd64 13.1-RELEASE-p1 amd64 http   2022-08-16 16:22:31 /usr/local/poudriere/jails/FreeBSD_13_amd64

A useful reminder that url_base should be set in poudriere.conf to where you would like to view build progress:

URL_BASE=https://poudriere.example.com/

A few more things to set-up before the build is started.

Securely sign packages

Packages can be signed using public/provate keys as identified in poudriere.conf:

PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key

Generate a private key using openssl

openssl genrsa -out myrepo.key 2048

Ensure it's owned by root and has read only permission

mv myrepo.key /etc/ssl/keys/
chmod 0400 /etc/ssl/keys/myrepo.key

Let poudriere.conf know where to find the private key

PKG_REPO_SIGNING_KEY=/etc/ssl/keys/myrepo.key

Generate the public key

openssl rsa -in myrepo.key -out myrepo.pub -pubout

Copy the public key to /etc/ssl/keys/:

cp myrepo.key /etc/ssl/keys/

Ensure it's readable by anybody

chmod 0444 /etc/ssl/keys/myrepo.key

Then let pkg (/usr/local/etc/pkg/repos/myrepo.conf) know where to find it:

pubkey: "/etc/ssl/keys/myrepo.pub",

Distribute your public key

There is a chicken and egg situation, in order to install a port you need to check the package with your public key before installation, but you need your packages installed to distribute your public key.

No doubt you have your own bootstrapping process when installing machines and creating jails/VMs. Distributing package public keys is included in infrastructure management to ensure consistency and facilitate change. Use your preferred infrastructure management / configuration management tool. Here are the instructions for Salt project:
Copy the public key for public distribution, it's treated like other public certificates and in this configuration certs.sls is included in top.sls.

cp myrepo.pub /usr/local/etc/salt/states/certs/
certs.sls
/etc/ssl/keys/myrepo.pub:
  file.managed:
    - source: salt://certs/myrepo.pub
    - makedirs: True

Package management

Package configuration files live in /usr/local/etc/pkg/repos/ and are named like the following:

/usr/local/etc/pkg/repos/myrepo.conf

The /usr/local/etc/pkg/repos/FreeBSD.conf is shown above, a local myrepo.conf might look like the following:

myrepo: {
  url: "https://packages.example.com/${ABI}-default",
  mirror_type: "none",
  enabled: yes,
  signature_type: "pubkey",
  pubkey: "/etc/ssl/keys/myrepo.pub",
  debug_level: 4,
}

The ABI is derived from /usr/bin/uname and looks like the following: FreeBSD:14:amd64.

After you have build your packages and are ready to switch from packages.freebsd.org to your own, ensure yours is set to enabled and edit /usr/local/etc/pkg/repos/FreeBSD.conf to "enabled: NO":

FreeBSD: { enabled: NO }

Start a package build

Start building your default packages with the bulk command:

poudriere bulk -j FreeBSD:14:amd64 -f /usr/local/etc/poudriere-default.packages

This will create the reference jail and start building the listed packages.

Making package build progress available

Any web server that can host static files can surface poudriere build progress.

Using apache as an example:

<VirtualHost *:443>
    ServerName packages.example.com
    ServerAdmin admin@example.com
    DocumentRoot /usr/local/poudriere/data/packages/
    SSLCertificateFile /path/to/example.com.crt
    SSLCertificateKeyFile /path/to/example.com.key
    SSLEngine on
    <Directory "/usr/local/poudriere/data/packages">
        Options None +FollowSymLinks
        SetHandler default-handler
        Require all granted
        AllowOverride None
    </Directory>
</VirtualHost>

Pointing your browser to your https://packages.example.com should present you with the latest builds for your jail and any build statistics.

Making packages available

After packages have been built, they need to be accessible across your estate.

If you only have one machine, or want to retrieve packages locally, the url can be a file location, for example your myrepo.conf might be:

url: "file:///usr/local/poudriere/data/packages/FreeBSD:13:amd64-default",

Most of the time you might want to use a generic one using the ABI:

url: "https://packages.example.com/${ABI}-default",

Any web server can be used to distribute packages, here is a simple Apache one:

<VirtualHost *:443>
    ServerName packages.example.com
    ServerAdmin admin@example.com
    DocumentRoot /usr/local/poudriere/data/packages/
    SSLCertificateFile /path/to/example.com.crt
    SSLCertificateKeyFile /path/to/example.com.key
    SSLEngine on
    <Directory "/usr/local/poudriere/data/packages">
        Options None +FollowSymLinks
        SetHandler default-handler
        Require all granted
        AllowOverride None
    </Directory>
</VirtualHost>

If your poudriere jails are not named according to the ABI "FreeBSD:14:amd64" then simply add aliases as necessary
Alias /FreeBSD:13:amd64 /usr/local/poudriere/data/packages/13amd64-default
Alias /FreeBSD:13:arm64 /usr/local/poudriere/data/packages/13arm64-default

You might wish to proxy requests for different architectures and protect by ip address using something like the following Apache virtual host.

<VirtualHost *:443>
    ServerName packages.example.com
    ServerAdmin admin@example.com
    UseCanonicalName On
    DocumentRoot /usr/local/www/example.com/public_html
    SSLCertificateFile /path/to/example.com.crt
    SSLCertificateKeyFile /path/to/example.com.key
    SSLEngine on
    SSLProxyEngine on
    SSLProxyCheckPeerName off
    <Location "/FreeBSD:13:amd64">
        <RequireAny>
                Require ip 2001:470:1f1d:362::5/128
                Require ip 192.168.1.0/24
        </RequireAny>
        ProxyPass "https://jail-amd64.example.com/"
        ProxyPassReverse "https://jail-amd64.example.com/"
    </Location>
    <Location "/FreeBSD:13:arm64">
        <RequireAny>
                Require ip 2001:470:1f1d:362::5/128
                Require ip 192.168.1.0/24
        </RequireAny>
        ProxyPass "https://jail-arm64.example.com/"
        ProxyPassReverse "https://jail-arm64.example.com/"
    </Location>
</VirtualHost>

Keeping up to date

It's essential to keep two ports trees. The first is the default ports tree at /usr/ports, the second is your default poudriere ports tree at /usr/local/poudriere/ports/default.

The first you keep up to date, perhaps weekly, so you know what is currently available. It's essential to read /usr/ports/UPDATING.

The second is your default poudriere ports tree that you will only update when you are ready to start a new build run. Remember building ports is under your control.

Upgrading a local ports tree in /usr/ports

If you don't have a working ports tree, then clone a new one

git clone https://git.FreeBSD.org/ports.git /usr/ports

Then update as necessary:

git pull --ff-only

You can check to see which ports can be upgraded

pkg version -v --index

xrandr-1.5.1                       =   up-to-date with index
xwayland-devel-21.0.99.1.211       <   needs updating (index has 21.0.99.1.262)

When you are ready to upgrade the following will upgrade your default poudriere ports tree:

poudriere ports -u

Check any information in /ports/UPDATING then run:

poudriere bulk -j [jail name] -f [your.packages]

Checking progress via your website https://poudriere.example.com/

When complete, you can upgrade using your normal configuration management, or your users can upgrade as normal with either a GUI or command line:

pkg upgrade

Summary

You now have a stable BSD build farm managing your applications. It's auditable and easy to manage.

Apps are continually changing and evolving. As they are now under your control, you'll find your own cadence for updating, building and distributing them.