unattended-upgrades on production servers: what it does and where it stops

Turn it on and check that it runs

Install the package and answer yes to the prompt:

sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

That writes two lines that switch on the daily list refresh and the daily upgrade run:

// /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

Two systemd timers do the work: apt-daily.timer refreshes the package lists and apt-daily-upgrade.timer installs, by default around 06:00 with up to an hour of random delay. Check the timers, do a dry run, and know where the logs are:

systemctl list-timers 'apt-daily*'
sudo unattended-upgrade --dry-run --debug
ls /var/log/unattended-upgrades/

What it installs

The list of allowed sources lives in /etc/apt/apt.conf.d/50unattended-upgrades. On Ubuntu the default is the release and -security pockets, plus ESM when you use Ubuntu Pro; regular -updates are commented out. Debian's default also takes stable point-release updates. Packages you pinned with apt-mark hold are skipped.

Settings worth changing on a production server

Put your own settings in a separate file instead of editing 50unattended-upgrades, so a package update never asks you to merge it:

// /etc/apt/apt.conf.d/52unattended-upgrades-local
Unattended-Upgrade::Package-Blacklist {
    "mariadb-server";
    "mysql-server";
};
Unattended-Upgrade::Mail "ops@example.com";
Unattended-Upgrade::MailReport "on-change";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Automatic-Reboot "false";

• Blacklist the services where an upgrade means a restart your customers notice, usually the database. Patch those in a window you choose.
• Mail only works with a mail transfer agent and mailx installed. Without it you get no report at all. Older releases use MailOnlyOnError instead of MailReport.
• Automatic-Reboot stays off here. When it is on, the server reboots at Automatic-Reboot-Time without checking whether anything is running that should not be interrupted.

What it leaves unfinished

• Some services keep the old library. After an OpenSSL or glibc update, a running process keeps the old copy in memory until it restarts. On Ubuntu, needrestart hooks into apt and restarts most affected services right after the run, at 06:00, whether that suits you or not. It defers the ones it considers risky, such as dbus, Docker and systemd-logind, and those keep running the old code. Set $nrconf{restart} = 'l'; in /etc/needrestart/conf.d/ if you would rather get a list than a restart. Debian does not install needrestart by default, so there nothing restarts.
• The new kernel is not running. It is installed and /var/run/reboot-required is written, but until someone reboots, the old kernel stays. On two of the three Ubuntu servers we checked while writing this, the running kernel was five and nine releases behind the installed one.
• Packages that need a decision are skipped. If an update wants to replace a config file you changed, the package is held back and the log says so, every day, until you deal with it: Package ubuntu-pro-client has conffile prompt and needs to be upgraded manually.
• Software apt does not manage is untouched. On DirectAdmin servers, CustomBuild compiles PHP, the web server, Exim and the database itself and excludes them from the package manager. Run da build versions (or ./build versions in /usr/local/directadmin/custombuild) to see what is out of date.

Two commands show what is left over after a run:

# A new kernel or libc is installed but not running yet
cat /var/run/reboot-required.pkgs

# Services still running old copies of updated libraries
sudo needrestart -r l

AlmaLinux and Rocky Linux: dnf-automatic

On AlmaLinux and Rocky Linux 8 and 9 the same job is done by dnf-automatic. Set it to security updates and to apply them, then enable the timer:

sudo dnf install dnf-automatic

# /etc/dnf/automatic.conf
[commands]
upgrade_type = security
apply_updates = yes

sudo systemctl enable --now dnf-automatic.timer

dnf-automatic restarts nothing: services keep old libraries until you restart them, and a new kernel waits for a reboot (needs-restarting -r tells you). On DirectAdmin servers the exclude= line in /etc/yum.conf keeps PHP, the web server, Exim and the database out of it, so CustomBuild remains your job.

A setup that works for a small fleet

• Keep automatic security updates on. A patched library waiting for a restart beats an unpatched one.
• Blacklist the database and patch it in a window.
• Every morning, read the reports and check reboot-required, services on old libraries and held-back packages.
• Schedule reboots yourself, one server at a time, and check the services after each one.

The morning check is the part that grows with every server you add: five minutes on one machine, an hour across twenty, every day.