Linux patch management for a small fleet: a process, a script and the tools

Start with a read-only fleet check

Before any tool, know where you stand. This script logs in to each server, simulates the upgrade and reports three numbers: pending updates, how many are security fixes, and whether a reboot is waiting. It changes nothing, so you can run it against production today.

#!/bin/sh
# morning-check.sh web01 web02 db01 ...
# Read-only: simulates the upgrade and reads state, changes nothing.
# On AlmaLinux and Rocky it needs passwordless sudo: as a normal user, dnf
# fails on repositories with repo_gpgcheck=1.
for h in "$@"; do
  printf "%-12s " "$h"
  ssh -n -o BatchMode=yes -o ConnectTimeout=10 "$h" '
    . /etc/os-release
    if command -v apt-get >/dev/null 2>&1; then
      sim=$(apt-get -s dist-upgrade 2>/dev/null | grep "^Inst ")
      all=$(printf "%s" "$sim" | grep -c "^Inst ")
      sec=$(printf "%s" "$sim" | grep -c -- "-security")
      [ -f /var/run/reboot-required ] && reboot=yes || reboot=no
    else
      list=$(sudo -n dnf -q check-update 2>/dev/null)
      if [ $? = 1 ]; then
        echo "$ID $VERSION_ID  dnf failed: check sudo, or run sudo dnf check-update"
        exit
      fi
      all=$(printf "%s\n" "$list" | grep -c "^[^ ]*\.[a-z0-9_]* ")
      sec=$(sudo -n dnf -q check-update --security 2>/dev/null | grep -c "^[^ ]*\.[a-z0-9_]* ")
      sudo -n needs-restarting -r >/dev/null 2>&1 && reboot=no || reboot=yes
    fi
    echo "$ID $VERSION_ID  updates $all  security $sec  reboot $reboot"
  ' || echo "unreachable"
done

Example output:

$ ./morning-check.sh web01 web02 app01 app02 db01 mail01
web01        almalinux 9.6  updates 412  security 268  reboot yes
web02        ssh: connect to host web02 port 22: Connection timed out
unreachable
app01        ubuntu 24.04  updates 51  security 0  reboot yes
app02        ubuntu 24.04  updates 34  security 1  reboot no
db01         rocky 9.5  updates 588  security 280  reboot yes
mail01       rocky 9.5  dnf failed: check sudo, or run sudo dnf check-update

Read it the way you would read a monitoring dashboard:

• Ubuntu with almost no security updates is normal: unattended-upgrades installed them overnight. What it leaves are regular updates and the reboot, see the unattended-upgrades guide.
• AlmaLinux or Rocky with hundreds means nothing installs security updates automatically. A fresh install has no dnf-automatic, so updates pile up until someone runs them.
• A failed check is the line to act on first. A repository with an expired or unimported signing key makes dnf stop, and a script that turned that into "0 updates" would tell you the server is fine. Ours did, in its first version.
• Unreachable is a finding too: a server you cannot check is a server nobody patches.

Look closer at one server

When a line stands out, these show what is behind the number. On Ubuntu and Debian:

apt list --upgradable
apt-get -s dist-upgrade | grep -- -security
cat /var/run/reboot-required.pkgs
sudo needrestart -r l

On AlmaLinux and Rocky:

sudo dnf check-update --security
sudo dnf updateinfo list --security
sudo needs-restarting -r     # exit code 1: reboot needed
sudo needs-restarting -s     # services running old libraries

A weekly routine that holds up

• Pick canaries. One server of each role (web, database, mail) gets updates a day before the rest.
• Security updates weekly, on a fixed day. Canaries first, then check the services they run: a page loads, the database answers, mail leaves the queue. The rest of the fleet the next day.
• Everything else monthly. Regular updates and point releases, such as AlmaLinux 9.6 to 9.8, change more and deserve a quieter day.
• Reboots as their own step. One server at a time, the same checks afterwards. A kernel that is installed but never booted protects nothing.
• Have a way back. Take a snapshot at your provider or hypervisor before larger runs. dnf can undo a transaction with dnf history undo as long as the old packages are still available; apt has no undo.

Both package managers already keep a record of what changed and when:

# AlmaLinux, Rocky
sudo dnf history list
sudo dnf history info 58

# Ubuntu, Debian
less /var/log/apt/history.log

The tools, and the size they fit

• Built-in automatic updates (unattended-upgrades, dnf-automatic). Free and already there. Security fixes per server, without an overview of the fleet and without reboots.
• Ansible. Agentless over SSH, so it fits a small fleet well. The dnf module can limit a run to security updates, serial: 1 gives you a rolling update, and the reboot module handles reboots. You write, schedule and maintain the playbooks yourself.
• Canonical Landscape. Patch and reboot management for Ubuntu, part of Ubuntu Pro. Ubuntu only.
• Foreman with Katello, the upstream of Red Hat Satellite. Built around the Red Hat family, with content views and errata. It is its own server with a database to run and update.
• Uyuni, the upstream of SUSE Multi-Linux Manager. Handles mixed fleets including Ubuntu, Debian and the Red Hat family, using Salt. Also its own server to run.
• RMM and patch SaaS such as Automox, NinjaOne or ManageEngine Patch Manager Plus. An agent on every server, a web dashboard, per-device pricing. Mostly built for Windows desktops, with Linux support next to it.
• Live patching (Canonical Livepatch, kpatch on RHEL, TuxCare KernelCare) applies selected kernel security fixes without a reboot. It shortens the gap; it does not replace the reboot.

On DirectAdmin servers none of these update PHP, the web server, Exim or MariaDB, because CustomBuild compiles them outside the package manager. See the CustomBuild guide.