Skip to content

Instantly share code, notes, and snippets.

@sirzerator

sirzerator/gentoo-introduction.md

Last active March 8, 2016 14:30
Show Gist options
  • Save sirzerator/7debabe7a5d9b962e1dc to your computer and use it in GitHub Desktop.
Save sirzerator/7debabe7a5d9b962e1dc to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gentoo-introduction.md

Gentoo

This document aims to cover the installation of a (simple) Gentoo system, the portage tree and the configuration of a personal overlay.

What is Gentoo ?

Gentoo (pronounced djen-too) is a source-based, rolling-release, Linux distribution whose primary goal is to provide a very flexible base on which to build a Linux system. Since every program is configured from the source, with only the specified options and compiler optimizations, it's possible to create a system perfectly tailored to your system and use case.

  • Portage

The "killer-feature" of Gentoo is most certainly its package manager, Portage... although it is not quite a package manager, yet much more than that. Portage is a collection of tools and scripts that manipulate ebuilds , the basic Gentoo "package". Ebuilds are nothing more than Bash scripts running in the Portage environment and describing how a program is to be patched, configured, compiled, installed as well as its dependencies. This is accomplished most notably with a combination of USE flags and KEYWORDS, a kind of tagging system for packages. Ebuilds can be compared to BSD ports, with dependency management and a lot more of options.

  • A meta-distribution

It can be argued that Gentoo is a meta-distribution, meaning that it can be used to reproduce any other distribution by specifically choosing the package versions, patches, configuration options, etc. It is, most certainly, a very "hands-on" distribution.

  • Speed gains ?

Not really. Gentoo is named after the Gentoo penguin, a fast Antarctica penguin. Soon after the first relase of Gentoo in 2002, it rapidly rose to fame (top 5 on distrowatch.com for several years) thanks to claims of significant speed gains. The enthusiasm of the beginnings rapidly faded when it was discovered that the gains were small and that Gentoo required, and still requires, more maintenance than other distributions.

Installing Gentoo

The process of installing Gentoo is pretty easy although a little more involved than most distributions. In this document, I'll try to get a Gentoo system running with as little steps as possible. These are the commands :

  • Setting up
fdisk /dev/sda
mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt/gentoo && cd /mnt/gentoo
wget http://path_to_stage3/stage3.tar.bz2
tar xvjpf stage3*.tar.bz2 --xattrs
mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mount -t proc proc /mnt/gentoo/proc && mount --rbind /sys /mnt/gentoo/sys && mount --rbind /dev /mnt/gentoo/dev
cp -L /etc/resolv.conf /mnt/gentoo/etc
chroot /mnt/gentoo /bin/bash
source /etc/profile
emerge-webrsync && emerge --sync
  • Installing
emerge vim gentoo-sources syslog-ng dhcpcd grub eix flaggie gentoolkit layman gpm  -va
cd /usr/src/linux && make menuconfig
make && make modules_install
cp arch/x86_64/boot/bzImage /boot/kernel-x.x-gentoo
cp .config /boot/config-x.x-gentoo
  • Configuring
passwd
vi /etc/conf.d/hostname /etc/conf.d/net /etc/ssh/sshd_config [and others]
ln -s /etc/conf.d/net.lo /etc/conf.d/net.x
rc-update add syslog-ng default && rc-update add sshd default && rc-update add sshd default && rc-update add net.enp0s3 default
vi /etc/fstab
echo "America/Montreal" /etc/timezone && emerge --config sys-libs/timezone-data
vi /etc/locale.gen && locale-gen
grub2-install /dev/sda && grub2-mkconfig -o /boot/grub/grub.cfg
  • Voilà!
reboot

Portage

Portage tree

The portage tree is a two-level directory structure. The first level is a category, the second level, a package name. Each folder contains one or more .ebuild files named as the folder with a version number, a files directory with patches and extra scripts and a few metadata files (Manifest, Changelogs, etc.) The default portage tree is located in /usr/portage .

A special directory, /usr/portage/distfiles, contains all downloaded source tarballs and patches. It can be emptied from time to time, it won't be done automatically.

Ebuilds are defined using the following syntax (an ebuild specification is commonly called an "atom"):

name
category/name
name-version (optionnaly prefixed by <, >, =, <= or >=)
name:slot
name::repository

A complete representation of an atom could look like this :

>app-editor/vim-7.4::sunrise

An extended format exist, used mostly in ebuilds. It allows to specify ranges, wildcards or negations : ~, * or !. It also provides a way to ask for the presence or absence of one or several USE flags : name[use,-notuse] .

In recent versions of portage, packages can be assembled in "sets" : just add a file in /etc/portage/sets with one atom per line and you have yourself a new set. They are accessible by prefixing @ to the set name. There are two legacy sets : @master and @system that can be called without the @ and contain the list of "selected" packages and the list of "system" packages, respectively.

USE flags, KEYWORDS, MASKS and make.conf

Portage packages are configured with USE flags. They define configuration options, dependencies or installation options. USE flags can be set (i.e. pulseaudio) or unset with a - sign (i.e. -pulseaudio).

USE flags are configured in the following ways, the latter taking precedence :

  1. Profile level USE flags (set within the selected profile /usr/portage/profiles/../make.defaults and others, c.f. eselect for profile selection)
  2. User-defined global USE flags in /etc/portage/make.conf
  3. User-defined per-package USE flags in /etc/portage/package.use
  4. USE environment variable for the current command.

The format of the package.use file is : atom [use flags]... .

echo "dev-lang/ruby xemacs -ipv6 >> /etc/portage/package.mask"

Packages also have "keywords" that serve to define supported or testing architectures. Keywords are simply an architecture short name, like amd64, prefixed with a ~ when the package is in the testing phase for that architecture. A Gentoo installation will have one primary architecture and keywords can be enabled per-package or globally.

  1. In /etc/portage/package.keywords for per-package keywords.
  2. With the environment variable ACCEPT_KEYWORDS for the current emerge run.

The format of package.accept_keywords files is : atom [keywords]....

echo "dev-lang/erlang ~amd64 >> /etc/portage/package.accept_keywords"

Finally, package masks can be defined. These will prevent a package from being installed. This is particularly useful if you set a range with a version number (i.e. >=gnome-base/gnome-desktop-2). Symmetrically, it is possible to unmask a package in the same way. Some packages, most notably VCS-based version (with version number 9999 by convention) are masked by default. Another common reason for masked packages, at profile level this time, is for deprecated packages or packages with known, unpatched security vulnerabilities.

  1. Masks are commonly defined in /etc/portage/package.mask
  2. Unmasks are defined in /etc/portage/package.unmask

The format of these files is just the atom.

All of the previously mentionned files in /etc/portage can be used as directories that contain files that will be parsed for patterns.

There are many other portage configuration files. Some are straightforward and will be explained by emerge as they are needed, like package.license.

Another interesting one is package.env that allows to define environment variables on a per-package basis. These will override make.conf for the selected package.

Last but not least, make.conf contains all global environment variables. This file is sourced by emerge and provides all general configuration options used throughout portage. There are too many to describe here and the most important are already present in the file by default. A common set of these environment variables are called USE_EXPAND and serve to define package-specific USE flags (c.f. www-servers/nginx and /usr/portage/profiles/desc).

emerge and ebuild

emerge is a Python script used to search in the Portage tree and optional overlays, install a package, resolve dependencies, etc. Its companion tool, ebuild, calls the different ebuild steps and can be used alone to manage or execute ebuilds.

A quick overview of emerge's most commonly used parameters (there's a ton) :

  • emerge --sync : Update the portage tree and configured overlays

  • emerge --search | -s : Search for a package by name (not very fast)

  • emerge --searchdesc | -S : Search for a package by name and by description (slow)

  • ^^ These can be replaced by eix for faster lookups and syncs

  • --pretend | -p : Will just print what it would do, but I prefer --ask in most cases.

  • --ask | -a : Like --pretend, but will interactively ask if it should continue or not.

  • --newuse | -N : Emerge package that have different USE flags than last emerge.

  • --deep | -D : Check the entire dependency tree instead of just immediate dependencies. Useful when updating or to resole dependency conflicts.

  • --tree | -t : Display ebuilds as a dependency tree (indented). Useful when trying to figure out why a package is being installed.

  • --keep-going : Don't stop on errors. Useful for large emerge sets.

  • --resume --skip-first : --resume restarts the last emerge set where it was interrupted. --skip-first skips the first ebuild of the last emerge set. Useful when not using --keep-going to skip over a broken package for later review.

  • --list-sets : Display a list of sets. There are several default, virtual sets.

  • --oneshot | -1 : Install a package without adding it to the world file.

  • --deselect : Remove a package from the world file.

  • --unmerge | -C : Uninstall (unmerge) a package.

  • --depclean : A little bit like apt-get autoremove, it will remove all packages that are not a dependency of a selected package

  • Some of these together for a comprehensive update : emerge world -vaNuDt

We'll look at the usage of ebuild later, when discussing overlays.

eix, a locate for portage trees

app-portage/eix provides a simple C utility that builds a binary cache of all relevant information in the portage tree, including overlays, and allows for quick lookup of any information. It is much more flexible than emerge --search or emerge --searchdesc and also provides useful options like --only-names -#, --installed -I and --in-overlay <overlay> .

It also comes with a tool called eix-sync that will sync the main portage tree and all installed overlays, independently of their configuration with regards to auto-sync. It then runs eix-update to rebuild the local database.

Once you start using eix, there is almost no good reason to go back to emerge or euse to search the portage tree.

flaggie, fun with flags

app-portage/flaggie is a small utility that simplifies managing flags (USE, KEYWORDS, MASKS, LICENSES, etc.). Its usage is very simple :

flaggie [<options>] [<global-actions>] [<packages> <actions>] [...]

Actions (global or per-package) can be either + to add, - to remove, % to reset and ? to query. A package is a regular atom definition.

It will automatically guess what type of flag(s) you are changing and add the required line(s) where they belong.

It also provides many options to clean the package.* files, but does not support the directory structure approach.

The gentoolkit, Gentoo's swiss army knife

app-portage/gentoolkit is a suite of tools aimed at managing and querying the portage tree and ebuilds. Just a quick glance at most useful ones :

  • equery provides many commands to query ebuilds.
Gentoo package query tool
Usage: equery [global-options] module-name [module-options]

modules (short name)
 (b)elongs               list what package FILES belong to
 (c)hanges               list changelog entries for ATOM
 chec(k)                 verify checksums and timestamps for PKG
 (d)epends               list all packages directly depending on ATOM
 dep(g)raph              display a tree of all dependencies for PKG
 (f)iles                 list all files installed by PKG
 h(a)s                   list all packages for matching ENVIRONMENT data stored in /var/db/pkg
 (h)asuse                list all packages that have USE flag
 ke(y)words              display keywords for specified PKG
 (l)ist                  list package matching PKG
 (m)eta                  display metadata about PKG
 (s)ize                  display total size of all files owned by PKG
 (u)ses                  display USE flags for PKG
 (w)hich                 print full path to ebuild for PKG
  • euse : a tool that lists USE flags present on the system with their state and allows to modify them. In practice, eix or flaggie are simpler and faster.
  • revdep-rebuild
    • Thanks to the virtual @preserved-rebuild set, the usefulness of revdep-rebuild has been significantly reduced.
    • It used to be an essential tool after any significant update. revdep-rebuild, standing for reverse dependency rebuild, will check all binaries in the path for missing dependencies, check to which package it belongs and rebuild it with --oneshot.
    • Recent versions of portage thankfully check for these reverse dependencies, keep required libraries and add the packages to a virtual set, @preserved-rebuild, for later rebuild and removal of the old dependency.

eselect, alternatives and toggles

app-admin/eselect is a little bit like alternatives on Debian-based systems, yet a little more. It works with anything that can be selected, activated, toggled or switched. From gcc and java default executables to opengl libraries, fonts configuration and distribution newsletter, it does it all.

eselect is composed of several modules, each one defining a series of actions. It's quite easy to explore from the command-line.

Portage overlays

Managing remote overlays with layman

app-portage/layman is a tool to manage portage overlays. It fetches a remote list of "official" (or rather approved) overlays and provides an interface for installing them on the system.

To use it, just add source /var/lib/layman/make.conf to your make.conf.

Creating a local gentoo overlay

A portage overlay is essentially a parallel portage tree. So, to begin, just create an empty folder with basic metadata definitions :

mkdir -p /usr/local/portage/{metadata,profiles}
echo 'demo' > /usr/local/portage/profiles/repo_name
echo 'masters = gentoo' > /usr/local/portage/metadata/layout.conf
chown -R portage:portage /usr/local/portage

Then tell portage that it exists :

mkdir -p /etc/portage/repos.conf
vi /etc/portage/repos.conf/local.conf

In /etc/portage/repos.conf/local.conf :

[demo]
location = /usr/local/portage
masters = gentoo
auto-sync = no

It used to be possible to simply define a PORTDIR_OVERLAY variable in /etc/portage/make.conf but this is getting deprecated (but still works fine).

Building a basic ebuild

This is the basic ebuild skeleton :

EAPI=5

DESCRIPTION=""
HOMEPAGE=""
SRC_URI=""

LICENSE=""
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

DEPEND=""
RDEPEND="${DEPEND}"

Let's say I want to use an old version of net-analyzer/netcat, available on Sourceforge. I first have to create the proper directory structure. I'll add it to a custom category called "phony" for demonstration purposes.

echo 'phony' >> /etc/portage/categories
mkdir -p /usr/local/portage/phony/netcat
vi /usr/local/portage/phony/netcat/netcat-0.5.0

EAPI=5

DESCRIPTION="Netcat"
HOMEPAGE="http://sourceforge.net/projects/netcat"
SRC_URI="mirror://sourceforge/netcat/${P}.tar.gz"

LICENSE="GPL2"
SLOT="0"
KEYWORDS="amd64"
IUSE=""

DEPEND=""
RDEPEND="${DEPEND}"

${P} is a special environment variable that will be replaced by the package name and version. There are other variants: ${PN}, just the name; ${PV}, just the version, etc.

Used like this, an ebuild is merely a wrapper for the usual ./configure && make && make install process with little added value. Let's say I checked at the configure options and want to add optional support for a feature. This is how I'd do it :

EAPI=5

DESCRIPTION="Netcat"
HOMEPAGE="http://sourceforge.net/projects/netcat"
SRC_URI="mirror://sourceforge/netcat/${P}.tar.gz"

LICENSE="GPL2"
SLOT="0"
KEYWORDS="amd64"
IUSE="-oldhexdump -oldtelnet"

DEPEND=""
RDEPEND="${DEPEND}"

src_configure() {
  econf \
    $(use_enable oldhexdump) \
    $(use_enable oldtelnet)
}

Now, let's say that, for some reason, I can't have both configure options at the same time and, if I have oldtelnet, I need telnet too.

Dependencies are achieved with the DEPEND and RDEPEND variables. These variables contain compile-time dependencies and runtime dependencies, respectively. These dependencies are described with a simple language of boolean expressions or atoms definitions.

USE flags exclusion can be defined in the src_prepare() function or in the REQUIRED_USE variable.

EAPI=5

DESCRIPTION="Netcat"
HOMEPAGE="http://sourceforge.net/projects/netcat"
SRC_URI="mirror://sourceforge/netcat/${P}.tar.gz"

LICENSE="GPL2"
SLOT="0"
KEYWORDS="amd64"
IUSE="oldhexdump oldtelnet"

DEPEND=""
RDEPEND="oldtelnet? ( net-misc/telnet-bsd )"

REQUIRED_USE="oldtelnet? ( !oldhexdump )"

src_prepare() {
  if use oldtelnet && use oldhexdump; then 
      die "Must use oldtelnet OR oldhexdump" # or better, "eerror"
  fi
}

src_configure() {
  econf \
    $(use_enable oldhexdump) \
    $(use_enable oldtelnet)
}

Now, let's clean up this mess and see emerge --depclean at work.

How does this all work?

Essentially, an ebuild is just a description of the installation process of the package (it's nothing more than a Bash script running in a controlled environment). The ebuild will be sourced by ebuild and the same general process will be applied: functions called in a certain order, checks made at a certain point, etc. The responsibility of a package or overlay maintainer is therefore to code the required functions override to install a package.

The process in which an ebuild is used goes as follow :

  1. emerge checks the dependencies, possible conflicts and determines an emerge order.
  2. For each package in order, ebuild is called with every command of the regular process :
  • setup : Runs system checks (all run at the beginning, may interrupt the emerge process or print a warning if not met)
  • clean : Cleans the temporary build directory (by default in /var/tmp/portage)
  • fetch : Downloads the required tarballs and patches (by default in /usr/portage/distfiles)
  • unpack : Extracts the sources in the temporary build directory. (src_unpack())
  • prepare : Run package preparation command (src_prepare())
  • configure : Configures the package (./configure basically, or src_configure())
  • compile : Compiles the package (make with all the bells and whistles, or src_compile())
  • test (optional, enabled with FEATURES="test", src_test())
  • install : Installs the package in a temporary directory, ready for merge (make install, or src_install())
  • qmerge : Really install the package (it will call pkg_preinst(), merge the files in the system, then pkg_postinst())
  • config : Runs post-installation configuration steps
  1. If it fails at some point, it stops and gives the error. The temporary build directory is not cleaned so you can check the build logs and figure out what went wrong.

A few other other commands exist, the most useful being :

  • manifest: Builds a Manifest file for every file referenced by the ebuild
  • merge: Just a shortcut for the whole process basically
  • unmerge: Removes the package by running pkg_prerm(), removes the files, then pkg_postrm(). This is made possible by the fact that emerge keeps track of every file it installs (which also prevents files conflicts).
  • package: Builds a binary package of the final result after install instead of merging it in the system.
  • There even is a rpm to build a RPM (RedHat) package.

Eclasses

The same behaviour may be used by several ebuilds. To simplify ebuild writing and avoid tedious repetition, portage provides a set of .eclass files. These files contain several functions that can be used to extend the behaviour of an ebuild. Eclasses are used with the inherit keyword like so :

inherit eutils autotools

eutils, for example, provides the epatch function that, no surprise, applies a patch. Some eclasses deal with VCS like git-r3 or subversion. These are commonly used in -9999 packages, i.e. sys-devel/llvm-9999.

Using userpatches

It's possible to add custom patches to an ebuild in any repository using a feature called userpatches (epatch_user). This works by adding your .patch in a directory with this format : /etc/portage/patches/${CATEGORY}/${PACKAGE}[-${VERSION}].

No need to create an ebuild for that. This feature is enabled by default in the standard src_prepare() function but may have been disabled by the package maintainer.

End

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment