Skip to content

Instantly share code, notes, and snippets.

@CodingKoopa

CodingKoopa/apt.md

Last active January 11, 2025 18:50
Show Gist options
  • Save CodingKoopa/3b30afe8c91e3950f6b124cd2abe3b6b to your computer and use it in GitHub Desktop.
Save CodingKoopa/3b30afe8c91e3950f6b124cd2abe3b6b to your computer and use it in GitHub Desktop.
Download ZIP
APT Repositories Explained
Raw
apt.md

Intro

To install and update packages on Debian, Ubuntu, or most derived distributions, you use APT, the Advanced packaging tool, to download the packages and their dependencies, which uses dpkg to install them.

These notes go into the downloading part, as I think it's easy to get overwhelmed by how the repositories work. I will eventually turn this into a page on my website.

Repository

A repository is where the packages come from. It provides binary packages for an arbitrary selection of architectures as well as source packages. Repositories are APT's data source. [https://wiki.debian.org/DebianRepository]

APT data sources can take different forms such as CD, but the one you likely use is HTTP(S), for downloading packages from a repository on the internet. The sources are configured in /etc/apt/sources.list in a one-line-style format in a format that we'll disect later.

A given server has one repository servicing the needs of a particular distribution, across releases. Since a mirror server may be doing mirroring for multiple distributions, it is likely (but not required) that the repository is located in a subdirectory of the server such as /debian/ or /ubuntu/.

Origin

Packages within this repository will have an Origin field indicating where the packages are coming from - not referring to this mirror server, but rather who put together the package. This is Debian for official packages, or the distributor otherwise [https://debian-handbook.info/browse/stable/sect.apt-get.html, "6.2.5. Managing Package Priorities "]. Though this article attempts to go through the hierarchy starting from the broader parts, it's worth noting this individual package detail because unattended-updates uses it for broadly categorizing packages.

Suite

Within a repository is a organizational unit for each release of the distribution. This unit has a few names depending on the context: [https://wiki.debian.org/DebianRepository/Format]

In the repository, the suites are located within the dists/ directory, e.g. /debian/dists/stable. With some Ubuntu-related exceptions, this organizational unit warrants its own section in UIs such as package search. That is, in navigating what a repository has from a website, they will usually list each suite that it has.

In practice, there are more than one suite for a distribution release. We will explore this once we have solidified the foundations of how APT works.

Component

A suite is broken up into different components, used for categorizing packages based on their libre purity and/or support guarantees.

The components are located in dists/<suite>/, e.g. /debian/dists/stable/main. However, unlike suites, these may not have their own section in UIs. In both Debian Package Search and Ubuntu Packages Search, packages from all components are pooled together, and packages that are not from main are labeled as such.

"Components" is a very universal term, though Debian can call them archive areas, and Ubuntu can (unfortunately) call them repositories, or channels. Unlike the different names for suites, though, these alternate terms are rare.

Release and Package Index

Within each suite is a release file, signed with GnuPG. Release is accomodated with Release.gpg, and the InRelease has the signature inside the file instead. The release file is comprised of hashes for the package indices, for verifying their integrity.

Within each component is another directory corresponding either with a particular architecture, or is source (or is one of the other exceptions that are out-of-scope). Along the directories themselves may be content indices, containing the files provided by packages.

Each architecture has a binary package index. The source directory has its own mechnanism. The package index contains metadata for every package in the component. The set of info for each package is a superset of the binary package control file, as described in the Debian Policy Manual.

There are two ways to get to the binary package index:

  • dists/<suite>/<component>/<architecture>/Packages.<ext>.
  • dists/<suite>/<component>/<architecture>/by-hash/<hash_type>/<hash>, using the hash from the release file.

In either case, the checksum of the package index will be verified by the package manager. The package index contains a Filename field containing a path within the archive (what we've been calling the suite), indicating where the actual package can be downloaded.

APT Process

The process of discovering and installing software starts with one of many package management tools. When it comes to frontends to APT, Debian suggests apt and aptitude for CLI, and Synapse for GUI. GNOME Software and Discover are DE-specifc GUI offerings. Ubuntu, on the other hand, provides the Ubuntu Software Center. Ubuntu also ships a GUI for configuring and reviewing updates.

These tools all use the lower-level apt- programs to install and update programs. The relationship between apt and apt-get is similar to that of adduser and useradd. In truth, the higher level frontends also could be using libapt directly, but the former assumption usually isn't harmful to make. [https://manpages.debian.org/stable/apt/apt.8.html] This article will proceed with the simplification that, by discussing apt-get, we are covering all of our bases. apt and related tools are all developed at https://salsa.debian.org/apt-team/apt.

It's the case for just about any Linux package manager that the package index has to be synchronized before packages can be downloaded. The "human-friendly" interface to do this is apt update. apt-get update is the lower level tool.

apt-get update goes through each of the APT data sources, in order. The sources are configured in /etc/apt/sources.list in a one-line-style format including: [https://manpages.debian.org/stable/apt/sources.list.5.html]

  • The type of the source, either deb (binary packages for the current architecture) and deb-src (source packages).
  • Optional repository options, such as a signing key.
  • The URI of the repository.
  • The suite within the repository to use.
  • Zero or more components within the suite to use.

The suite and components may contain slashes. What matters is that there is a Release file after concatenating the repository, suite, and each component.

For each of the data sources, apt-get update downloads the package index to /var/lib/apt/lists/. This directory comprises APT's package cache, and is its source of truth for downloading packages. [https://debian-handbook.info/browse/stable/sect.apt-get.html, "6.2.1. Initialization "] This database can be queried using apt search, apt show, and in more detail with apt-cache [https://debian-handbook.info/browse/stable/sect.apt-cache.html, https://manpages.debian.org/stable/apt/apt-cache.8.html]

A package can exist in more than one APT data source, potentially even of the same version. To automatically decide which one to use, APT prioritizes, in order:

See the apt_preferences manpage for a more rigorous definition. You can run apt-cache policy <package> to see what APT sees, for a particular package. This includes each source for the package, and the priorities and versions.

Additionally, you can manually select a data source as described in the apt_preference manpage. When the "target release" is explicitly given, and a package from it is installed, it defaults to a priority of 990. When it comes to getting it installed in the frist place, though, this "target release" overrides general priorities, but not specific package priorities.

Overall, having added multiple APT data sources, there are several ways to influence how APT treats a package:

  • Configuring a target release to make APT prioritize that distribution over others (withstanding package pins).
  • Pinning a package or a suite.
  • Holding a package, preventing it from being upgraded.
  • Installing a specific package version by running apt-get install <package_name>=<package_version>. Couples well with holding, to prevent automatic upgrades.
  • Installing a package from a specific suite by running apt-get install <package_name>/<suite>-backports. [https://help.ubuntu.com/community/UbuntuBackports#Manually_Installing_Packages_from_Backports].

For most of these techniques (all but the first), even if a non-default version of a package is installed, the default versions of the dependencies will be used. That is, if you install backport without pinning, the non-backport dependencies will be used if possible.

Once APT has decided which package to install, it then has to resolve its dependencies, and then use dpkg to install them. dpkg will resolve any conflicts that arise with existing files or modified configuration files. This is all outside of the scope of this article, though.

Having established how APT interacts with repositories, we can go into the specifics of how Debian and Ubuntu uses them.

APT in the Wild

Debian Repositories

Debian has the following repositories [https://wiki.debian.org/SourcesList]:

Of these sources, Debian systems tend to default to <release> + <release-security> +<release-updates> (where there shouldn't be any significant tie breaking done by the order) from the current releases. <release> + <release-security> is also well-supported. Backports has to be manually enabled, and has less support guarantees. It's possible to use the stable aliases rather than the specific codenames, but not recommended as this will perform a distribution upgrade, seemingly randomly once released.

To help solidify what the pieces do, the lifespan of a package is very roughly like this:

  • Upstream publishes source code.
  • Debian packager creates package.
  • Debian package uploads package to Debian Unstable.
    • If security updates occur here, they are addressed directly through package updates.
  • Package is promoted to next stable (Debian Testing).
  • Package is promoted to stable. Initial point release is made.
  • Stable updates are pulled from upstream and pushed to stable-proposed-updates.
  • Some of proposed updates are pulled into stable-updates.
  • If a security issue occur here, the Debian Security Team patches the package and uploads it to the Debian Security repository.
    • The pacakge version chosen to be patched is the latest out of stable, stable-security, stable-updates, and stable-proposed-updates. This is necessary so that:
      • Users with stable-updates do not experience regressions in non-security bugs.
      • The patched package can be integrated into the next point release without any changes.
      • The security team does not have the burden of maintaining patches for every suite.
    • With this said, "The most important guideline when making a new package that fixes a security problem is to make as few changes as possible.".
    • The package updated by the Security Team in stable-security is automatically uploaded to stable-proposed-updates.
  • When the time comes for a new point release, everything from stable-proposed-updates becomes the new stable.

For the purposes of the changelog, the Debian Security Team signs off the distribution as <release>-security for security fixes. When it comes to stable-updates, though, since they originate as proposed updates destined for the next point release, in the changelog they are simply <release>. In other words, you will never see -updates in a changelog file.

Debian Package Examples

Let's look at some examples of what APT sees, on my Debian Buster virtual machine with the suites buster, buster-updates, and buster-security. Starting with a simple one:

$ apt-cache policy vim
vim:
  Installed: (none)
  Candidate: 2:8.1.0875-5+deb10u2
  Version table:
     2:8.1.0875-5+deb10u2 500
        500 http://deb.debian.org/debian buster/main amd64 Packages

Vim has not had any stable updates or security fixes, so there is no Vim entry in the package index for buster-updates. There is only the Vim version that shipped with the last Buster point release.

When it comes to stable updates, a nice example is antivirus data:

$ apt-cache policy clamav
clamav:
  Installed: (none)
  Candidate: 0.103.6+dfsg-0+deb10u1
  Version table:
     0.103.6+dfsg-0+deb10u1 500
        500 http://deb.debian.org/debian buster-updates/main amd64 Packages
     0.103.5+dfsg-0+deb10u1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages

There is the version that was shipped by the point release, and then the newer package that is in the update suite. APT will prefer the package with the higher version.

Thunderbird, on the other hand, exemplifies how extreme the security suite can be:

$ apt-cache policy thunderbird
thunderbird:
  Installed: (none)
  Candidate: 1:91.12.0-1~deb10u1
  Version table:
     1:91.12.0-1~deb10u1 500
        500 http://security.debian.org/debian-security buster/updates/main amd64 Packages
     1:78.14.0-1~deb10u1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages

It seems that, in order to fix the exploits, the Security Team decided to execute a huge version bump.

Ubuntu Repositories

Ubuntu has the following repositories:

release + release-security is considered the minimum for a supported Ubuntu system. release + release-security + release-updates is the default for Ubuntu systems.

Like we did for Ubuntu, let's go through the life of a package:

  • Upstream publishes source code.
  • Debian packager creates package.
  • A new release suite is created from the latest versions, and aliased to devel. Ubuntu Developers upgrade their packages to a new upstream release. [https://wiki.ubuntu.com/UbuntuDevelopment/ReleaseProcess]
    • By default (for most packages), this will be synced from Debian. [https://wiki.ubuntu.com/UbuntuDevelopment/PackageArchive#Syncing_and_Merging]
    • Otherwise, if Ubuntu modified the package (with ubuntu in the version), either:
      • The package is gotten back in sync.
      • A merge is performed and the divergence remains.
    • "This phase can be said to end when all packages have been brought up to date at least once, and the result has been sufficiently stabilized to produce the first milestone, with installable CD images."
  • The development release undergoes several freezes, until it becomes a release candidate. [https://wiki.ubuntu.com/UbuntuDevelopment/ReleaseProcess#Stabilization_.28Freeze.29]
    • Unlike Debian, Ubuntu has regular biannual releases. As a result, many freezes are time-based.
  • The final release is made, and the first point release is made.
  • Stable release updates are made and pushed to <release>-proposed.
  • After sufficient testing, the proposed updates are pulled into <release>-updates in phases, without rebuilding.
    • If a security issue occurs here or in the point release, the Security Team will patch the latest package from <release> + <release>-security + <release>-updates, and upload it to <release>-security.
      • It was mentioned before, but: After about a half hour, this will be mirrored to <release>-updates for better availability.
  • When a point release is scheduled:
    • Pushing updates to <release>-proposed is discouraged to allow all the existing updates to be flushed for <release>-updates.
    • The point release is published with the latest packages from <release> + <release>-security + <release>-updates.

Ubuntu Package Examples

Let's go through some apt-cache examples on Ubuntu. These will be more complicated than Debians', for less redundancy.

First, a stable release update:

$ apt-cache policy unattended-upgrades
unattended-upgrades:
  Installed: 1.1ubuntu1.18.04.14
  Candidate: 1.1ubuntu1.18.04.14
  Version table:
 *** 1.1ubuntu1.18.04.14 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/main amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/main i386 Packages
        100 /var/lib/dpkg/status
     1.1ubuntu1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic/main amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu bionic/main i386 Packages

There's not much new here, other than this is a case where I actually have the package installed. Accordingly, there is a source for the package with priority 100; that's the one that I have installed.

We've seen ClamAV antivirutes updates be delivered as stable, non-security updates. In this case, though, there has been a security update issued:

$ apt-cache policy clamav
clamav:
  Installed: (none)
  Candidate: 0.103.6+dfsg-0ubuntu0.18.04.1
  Version table:
     0.103.6+dfsg-0ubuntu0.18.04.1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     0.99.4+addedllvm-0ubuntu1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic/main amd64 Packages

We know that bionic-security gets mirrored to bionic-updates after a little bit. So, we can draw from this that:

  • The current Ubuntu Bionic Beaver point release ISO comes with ClamAV 0.99.4.
    • This is neither upstream nor Debian's ClamAV 0.99.4, as indicated by the ubuntu string in the version.
  • A security issue was discovered, which affects ClamAV 0.99.4.
    • This does affect Debian too, but it seems to have not been delegated to the Security Team.
    • To find this, I just looked up "ubuntu clamav cve" to get to the security notice.
  • The security team pushed ClamAV 0.103.6+dfsg-0ubuntu0.18.04.1 to bionic-security.
  • The updated package was pushed to bionic-updates, propagating to mirrors including the one I am using in this example.

I can also tell that this package last got a security update by looking at the Ubuntu Packages Search, noticing that there is a security version, and it is the same version that is in the update suite.

Now, a mixed example:

$ apt-cache policy librte-vhost17.11
librte-vhost17.11:
  Installed: (none)
  Candidate: 17.11.10-0ubuntu0.1
  Version table:
     17.11.10-0ubuntu0.1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/main amd64 Packages
     17.11.9-0ubuntu18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     17.11.1-6 500
        500 http://mirrors.digitalocean.com/ubuntu bionic/main amd64 Packages

This package received a security fix, and then an update fix that will be based on that. If this package gets another security fix, it will look something like this:

     17.11.11 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     17.11.1-6 500
        500 http://mirrors.digitalocean.com/ubuntu bionic/main amd64 Packages

The release version stays the same, and the security version is arbitrarily higher so that it will be used. 17.11.11 is a version with the security fix that was based on 17.11.10-0ubuntu0.1 from bionic-updates, because that was the latest package update.

Now, a weirder one:

libgnat-8-dbg-hppa-cross:
  Installed: (none)
  Candidate: 8.4.0-1ubuntu1~18.04cross1
  Version table:
     8.4.0-1ubuntu1~18.04cross1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/universe amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/universe i386 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages
     8.2.0-1ubuntu2~18.04cross1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/universe amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu bionic-updates/universe i386 Packages
     8-20180414-1ubuntu2cross1 500
        500 http://mirrors.digitalocean.com/ubuntu bionic/universe amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu bionic/universe i386 Packages

There is an older version of this package in bionic-updates that will not be defaulted to in any circumstances. I guess that old packages aren't removed immediately from the repo.

Debian and Ubuntu Components

Now for something a little easier. Within the official suites, Debian has the following components: [https://www.debian.org/doc/debian-policy/ch-archive.html#archive-areas, https://wiki.debian.org/SourcesList#Example_sources.list]

  • main, for free (as in freedom) software that only requires other main packages, as determined by the Debian Free Software Guidelines.
  • contrib, for free software with dependencies not in main or contrib (either as a package dependency, or by being an accessory).
  • non-free, for non-free software.

contrib and non-free are not official parts of the Debian distribution, and are not maintained by the Security Team. [https://www.debian.org/security/faq#contrib]

Ubuntu has the following components: [https://ubuntu.com/about/release-cycle, https://wiki.ubuntu.com/SecurityTeam/FAQ#Standard_Support, https://help.ubuntu.com/community/Repositories#Components, https://help.ubuntu.com/community/Repositories/Ubuntu#The_Four_Main_Repositories]

  • main, Canonical-maintained free packages.
  • restricted, Canonical-maintained non-free packages (mostly just proprietary drivers).
  • universe, community-maintained free packages.
  • multiverse, community-maintained non-free packages.

The Security Team only maintains main and restricted, and for the duration of the standard support. [https://wiki.ubuntu.com/SecurityTeam/FAQ#Standard_Support]

Upgrades

https://help.ubuntu.com/community/AutoWeeklyUpdateHowTo

unattended-programs is a Python program that uses APT to install upgrades without user intervention. By default, once enabled it will only upgrade packages from the <release>-security suite. Like with apt upgrade, it will not remove packages to satisfy a transaction. This is documented by Debian at https://wiki.debian.org/UnattendedUpgrades, and Ubuntu at https://help.ubuntu.com/community/AutomaticSecurityUpdates. This supersedes the older technique of running a cron job. [https://help.ubuntu.com/community/AutoWeeklyUpdateHowTo]

When it comes to regular package upgrades, Ubuntu has its Software Update GUI.

As for distribution upgrades, Ubuntu has an upgrade GUI and upgrade script do-release-upgrade, both backed by update-manager [https://askubuntu.com/a/409556/497651]. Debian provides manual instructions, but a Debian user suggests that automation is possible, perhaps by adopting Ubuntu's work.

Conclusion

I started these notes to ascertain how Ubuntu's update channels work. The software update interface makes it seem like, when you only enable the security update channel, you only get security updates. However, after sifting through all the documentation, this isn't always the case: once a non-security bug fix has been pushed, future security bug fixes will be based on that latest package. Only using the security update channel does not control *which- updates (out of <release>-security, <release-updates>, and <release-proposed> by proxy) you get, but rather *when- you get them. The same goes for Debian.

This info is useful to know when configuring unattended-upgrades, as it defaults to only using the security channel. It's my opinion that most users and system administrators are not given the precise information they need to make this decision, something which I think plagues the landscape of system administration on Ubuntu. I polished this into a longer form of writing to help combat cargo-cult administration by encouraging curiosity about the underlying systems.

More info

I have linked to the specific resources that I have used, but there are some more general pages that provide a good overview, including some links that I did not use:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment