packer-configs

A collection of Packer templates for various uses. These were written in a way that should help you understand packer by looking at how they work, and expanding on them.

Cloning

Important

This project uses the ansible-roles repo and its submodules. Use --recursive to clone this project with submodules.

To clone everything, recursively over SSH:

# Using SSH authenticated to your GitHub account
git clone git@github.com:straysheep-dev/packer-configs.git
cd ansible-configs
# Initialize all submodules, you'd do this to pull the latest changes as well
git submodule update --init --checkout --recursive
git submodule sync --recursive

To clone everything, recursively over HTTPS:

# Point to HTTPS instead of SSH, unauthenticated without a GitHub account
git clone https://github.com/straysheep-dev/packer-configs.git
cd ansible-configs
git -c url.https://github.com/.insteadof=ssh://git@github.com/ \
    -c url.https://github.com/.insteadof=git@github.com: \
    submodule update --init --checkout --recursive \
    git submodule sync --recursive
# Set the change for each submodule at the project level after initializing
git submodule foreach --recursive \
    'git config url.https://github.com/.insteadof ssh://git@github.com/
    git config url.https://github.com/.insteadof git@github.com:'

If you want to update just the ansible-configs submodule itself to the latest commit:

# Update the ansible-configs submodule
git submodule update --remote

If you want to pull the role files down locally to use in packer builds:

# Run the same command one layer down, within ansible-configs,
# to pull the latest role files locally for use with packer
git submodule foreach 'git submodule update --remote'

Note

The status for ansible-lint is from the straysheep-dev/ansible-configs repo. These roles will always be sync'd from there, unmodified. The CI workflow here appeared to have issues even with the working_directory: argument before these roles were moved to a submodule.

Resources

A list of resources used to create these templates.

• Packer QEMU Builder
• Packer QEMU Plugin UEFI Boot
• Packer Local Variables
• Packer Input Variables
• Packer Conditional Expressions
• Cloud-init Autoinstall Reference
• Cloud-init Data Source Syntax
• Cloud-init: Drive with labeled filesystem
• Cloud-init user-data Reference
• Cloud-init ISO Disk vs HTTP Server
• Debian Secure Boot VM Settings
• Ubuntu Server UEFI-Enabled Templates
• chef/bento Packer Templates
• Packer Guides: Unattended Installs
• github.com/canonical/autoinstall-desktop
• github.com/canonical/packer-maas Templates
• boot_command Keys
• boot_command Examples
• kali-vagrant Build Scripts
• Ubuntu Docs: Virtual Machine Manager
• Support added for internal snapshots of UEFI VM's in virt-manager
• git-scm.com: Submodules

Getting Started

ISO and IMG File Management

• Packer will write ISO's to ~/.cache/packer/
• You can move these to a more standard path like ~/iso or ~/src/my-project/ubuntu-2204/iso/
• This means you can even tell packer to save ISOs it fetches to that path instead of cache
• You can also tell packer to try a local file first before fetching the remote file
• If you intend to use virt-manager, the path for those files is /var/lib/libvirt/

Using Packer

• The "source" .pkr.hcl file is the base file that's the minimum necessary to run a build
• locals.pkr.hcl is for ingesting variables, processing conditionals or expressions, and holding static values that don't change
• variables.pkr.hcl declares anything that can have a modified input
• *.pkrvars.hcl files are where the values for each unique build can exist
• Use packer only for what's necessary (base install and build)
• Use provisioners like the Ansible or Shell provisioners for complex configuration after the build completes
• This reduces issues Packer can run into, and makes your templates more portable

Essential Commands

Manually install plugins.

# List installed plugins
packer plugins installed

# Install plugins
packer plugins install github.com/hashicorp/ansible
packer plugins install github.com/hashicorp/qemu

Initialize a packer template. This downloads any missing plugins described in the packer {} block to your local machine. Review plugins before executing this.

cd /path/to/my-packer-project
packer init .

Format the packer templates. Rewrites HCL2 config files to canonical format. Run this after making edits to your templates or as part of CI/CD.

packer fmt .

Validate the packer templates. Run this after making edits to your templates or as part of CI/CD.

packer validate .

Start a build. This will launch whatever is necessary to build the template, and may take some time. In a GUI you'll see the machine launch and build live.

packer build .

Debugging Common Issues

On-Error Options

-on-error=<OPTION> allows you to configure what packer does when it encounters an error. By default it will run the cleanup option, which deletes all temporary files and the VM image it was building. This makes it impossible to examine what went wrong on the VM while developing a template, but is ideal in a CI/CD scenario.

-on-error=ask presents a prompt and waits for you to decide to clean up, abort, or retry the failed step.

Debugging on the VM

Use Alt+F3-6 to open a shell on the build VM if you have a GUI through QEMU. This is the easiest way to read logs and run any diagnostic commands.

Log Files

• Ubuntu: /var/log/installer/
  • sudo tail -F /var/log/installer/subiquity-server-*.log can provide more information than the default output

Build Stalls During OS Install

If a build relies on networking and appears stuck, you won't have an easy time diagnosing issues from the host.

Tip

Even with constrained resources on a build VM, a nested packer VM being built should still complete the OS install within ~10 minutes depending on the CPU speed. This will vary, but has been consistent across both Debian family OS's and Windows Server.

• Attempt to ping or curl endpoints it's trying to reach
• The host may be filtering packets (unintentionally) it's a good idea to restart the network renderer and your DNS resolver
• Use tcpdump on the ad hoc QEMU interface and network (10.0.2.2)

Boot Commands

This is possibly the most esoteric piece of packer if you're just getting started and haven't read how to write these yourself. Packer automates building VMs. This sometimes includes the literal key presses it takes as if you were building one manually. Packer emulates these key presses so the entire process is fully automated. Packer's boot command syntax is summarized in the link as:

The boot command "typed" character for character over a VNC connection to the machine, simulating a human actually typing the keyboard.

Use that guide to help you write and modify your own boot commands.

Packer and Ansible

Ansible often requires a credential, whether it's to a vault or for sudo to execute elevated tasks. There are two easy ways to do this in a more secure way than hard-coding a plaintext passphrase.

1. A "password file" containing the user's sudo password in raw text, on one line. Point to this file with --become-password-file=./ansible/pwfile.

build {
  name    = "my-vm"
  sources = ["source.qemu.my-vm"]

  provisioner "ansible" {
    playbook_file = "./ansible/playbook.yml"
    extra_arguments = [
      "--become-password-file=./ansible/pwfile"
    ]
  }
}

2. A "password file" containing a the password to an Ansible vault. Do the same as above, pointing to a file containing just this string, on one line.

build {
  name    = "my-vm"
  sources = ["source.qemu.my-vm"]

  provisioner "ansible" {
    playbook_file = "./ansible/playbook.yml"
    extra_arguments = [
      "--extra-vars",
      "@./ansible/vault.txt",
      "--vault-password-file=./ansible/pwfile"
    ]
  }
}

As better ways to handle credentials are tested across platforms, and with CI/CD, they will be added here.

Ansible hostfwd

You can use the -netdev user,id=net0,hostfwd=tcp::2222-:22 \ port forward to reach the VM through the host to run additional Ansible roles after the build completes.

Here's an example inventory file using the host 2222:22 port forward:

packergroup:
    127.0.0.1:
        ansible_port: 2222
        ansible_user: kali

You'll need a public key on the packer machine, or sudo apt install -y sshpass on the Ansible / Packer host.

Run Completed Builds with QEMU

Modern Linux distros support EFI boot as well as the option to run with Secure Boot functionality if you enable it. Windows 11 requires Secure Boot by default, and for some built-in threat mitigation features to work.

These resources cover what you need to execute kvm or qemu-system-<arch> to run a machine that will be installed or was installed with Secure Boot.

• superuser/qemu-kvm-uefi-secure-boot-doesnt-work
• Debian Secure Boot VM Settings

Practically what you need are 3 items: the Q35 chipset, the EFI firmware ROM (read-only) and the EFI variables unique to that machine (writable). The EFI variables file is typically copied to a place for use with that VM from /usr/share/OVMF/OVMF_VARS_*. Which file you need depends on which EFI ROM you used to provision the VM.

• -machine q35,smm=on,accel... handles the chipset, enables smm for Secure Boot, and tells the host what acceleration to prefer based on what's available, in that order
• -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.secboot.fd points to the EFI ROM
• -drive if=pflash,format=raw,file=efivars.fd points to a local, writable, copy of the EFI variables.

This all assumes you cd into a build/ directory that contains the .qcow2 VM image, and the efivars.fd file that machine can write to. Boot that machine with the following command:

cd build/ && \
kvm -no-reboot -m 4096 -smp 4\
  -cpu host \
  -machine q35,smm=on,accel=kvm:hvf:whpx:tcg \
  -global driver=cfi.pflash01,property=secure,value=on \
  -device virtio-net-pci,netdev=net0 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -object rng-random,filename=/dev/urandom,id=rng0 \
  -drive file=ubuntu-2204,format=qcow2,if=virtio \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.secboot.fd \
  -drive if=pflash,format=raw,file=efivars.fd \
  -vga virtio \
  -display gtk

Run Completed Builds with virt-manager

Move the VM disk image and EFI variables into the correct virt-manager paths.

sudo cp build/ubuntu-2204-desktop /var/lib/libvirt/images/ubuntu-2204-desktop.qcow2
sudo cp build/efivars.fd /var/lib/libvirt/qemu/nvram/ubuntu-2204-desktop_VARS.fd

Set the ownership to libvirt-qemu:kvm (this was done on an Ubuntu host, your user:group may be different).

sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/ubuntu-2204-desktop.qcow2
sudo chown libvirt-qemu:kvm /var/lib/libvirt/qemu/nvram/ubuntu-2204-desktop_VARS.fd

Now open the virt-manager GUI.

• Set Q35 for the CPU chipset
• Select UEFI firmware and choose the OVMF_CODE_4M.secboot.fd ROM

You will need to manually edit the <os> element of the XML config under CPU (or Overview).

The <nvram> line is what you'll need to add. Change the NVRAM path to point to the variables file you just copied over.

  <os>
    <type arch="x86_64" machine="q35">hvm</type>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/OVMF/OVMF_CODE_4M.secboot.fd</loader>
    <nvram template="/usr/share/OVMF/OVMF_VARS_4M.ms.fd">/var/lib/libvirt/qemu/nvram/ubuntu-2204-desktop_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>

Apply, then choose "Begin installation". This will boot the VM. You will likely have a blank window after the initial QEMU boot splash screen. The easiest thing to do here is click into the blank window and press the esc key to cycle through the console output until you get the GUI. You can also try to close / reopen the VM window. If this doesn't fix the video output, shutdown the VM with the CLI or virt-manager tools and then close / reopen the window. You should have video output at this point.

Another issue you may encounter is when you install a desktop image onto a server VM, the networking configuration may need revised to work automatically as expected. This happened in the case of the first config in this repo, ubuntu-2204-desktop.

Automate with virt-install

Inside each packer template folder is an import-to-virt-manager.sh file that will automate the steps above for you.

• RedHat: Creating Guest VMs with virt-install
• RedHat: Comparison of virt-manager and virt-install Options

Proxmox Usage

Note

These templates are currently built to work with the qemu builder, not the proxmox builder. However, this does not prevent you from building on Proxmox with a dedicated "build" user that can run packer.

That guide uses the Proxmox packer integration and ultimately creates a Proxmox account with those groups and privileges in mind. To simply create a dedicated local build user as if this was a normal Debian server:

# Create the user, they'll have a $HOME folder
useradd "builder"

# This account needs access to the kvm group for QEMU usage
usermod -aG kvm builder

You'll need to install Packer, and install Ansible on Proxmox itself so your build user can create the machines.

# Install Packer, system-wide as root
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install packer

# As the builder user
su builder

# Install Ansible
python3 -m pip install --user ansible

Note

While it's better to use a dedicated build VM, almost as a GitHub Actions runner, this guide is considering the fact that you have to provision that build VM to even get started. This is one option to do that.

One major difference when working directly on Proxmox is the EFI variables for:

  efi_firmware_code = "/usr/share/OVMF/OVMF_CODE_4M.secboot.fd"
  efi_firmware_vars = "/usr/share/OVMF/OVMF_VARS_4M.ms.fd"

These will need to point to:

  efi_firmware_code = "/usr/share/kvm/OVMF_CODE-pure-efi.fd"
  efi_firmware_vars = "/usr/share/kvm/OVMF_VARS-pure-efi.fd"

This is a temporary work-around, as the ovmf package in Proxmox isn't as up-to-date as the one you'd find in Ubuntu (which implies it's not the right answer to this problem). The /usr/share/kvm/ directory has numerous UEFI and BIOS files to review. Once a more complete solution has been developed this section will be updated with those details.