CommitThis/customising_mion.md

## customising_mion.md

      
    Raw
  

              customising_mion.md
            
          
    Customising mion

It is possible to create a mion image with your own software and features.
This article will take you through some basics to get you started.
Introduction

mion is an operating system and build system based on the Yocto Project tooling.
There are a few key terms you will need to be aware of:

BitBake is the tool and build system used to execute builds;
A recipe contains instructions, patches and files required to build and
install a given feature or software package;
A bbclass (BitBake class) is the skeleton of a recipe, providing common
instructions and variables, and can be inherited to form a complete recipe;
A layer is a collection of recipes, classes and other configurations;
A package is the result of a recipe build.
package-group is a list of packages that can be installed or packaged
together.

The Yocto Project/OpenEmbedded is a community and organisation that
maintains BitBake, core layers, and recipes, and mion is a Linux
distribution built from a collection of core layers and recipes, as well
as those that that are platform specific.
Quick Links

While this document is a short guide for some basic activities, there is
extensive documentation on the Yocto Project website. This will be referred
to throughout this page, however, some quick links follow:

Yocto Project Documentation
What I wish I’d known about Yocto Project
Overview and Concepts
Board Support Package Development Guide
BitBake User Manual
Yocto Project Mega-Manual
OpenEmbedded Layer Index

mion's Directory Structure

The project's directory structure looks intimidating at first, but is actually
relatively straightforward.
mions's base git repository contains
the base layers, utility scripts and some configuration files. This repository
in and of itself is not enough to build an image.

Layer directories can be distinguished by having the meta- prefix.

The recipes used to actually generate an image reside in meta-mion, which is
a layer that has recipes for ONIE images, which also include the base packages
that are installed, e.g., bash, glibc, ipmitool etc.
This is, however, not yet enough information to build an image. For one, each
target system has their own architecture, and we haven't gotten to any machine
specific information yet!
This is where the BSPs (Board Support Packages) come in. These are typically
either layers that contain recipes and configurations for specific machines.

Note: The term BSP is overloaded. Both Yocto Project and the Intel SDE
have their own concept of what they are. Yocto Project BSPs contain
information for building images, where a vendor's BSP for the Intel SDE
contain software to get the device's Tofino ASIC working.

Supported BSPs are located in the mion/meta-mion-bsp directory, which in itself is a collection of layers representative of each
manufacturer.
Taking the APS (née Stordis) devices, the vendor layer is located at
mion/meta-mion-bsp/meta-mion-stordis. This then has recipes for packages, as well
as configurations unique to that device.
Taking the BF6064X-T as an example, the configuration that specifies the
target architecture (as well as console/tty settings) can be found in
stordis-bf6064x-t.conf.
Navigating through this layer, you will find concrete recipes for installing
ONLP, the Intel SDE and associated (Barefoot) BSP, extra kernel modules, kernel
defconfigs.
And finally, to close the loop, these recipes will typically depend on other
recipes and classes found in the layers we have at the top-level of the project!
Adding Layers to the Project

To get access to additional third-party recipes, classes and package-groups, you
will want to add extra layers to your project. There are two steps:

Clone or copy the layer's folder to the root of the mion project,
Make BitBake aware of the layer by adding it to the layer configuration.

The layer configuration file is bblayers.conf and is found in mion/build/conf/.
The default mion layer configuration follows:
# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "7"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

MIONBASE = "${@os.path.abspath(os.path.join("${TOPDIR}", os.pardir))}"

BBLAYERS = " \
  ${MIONBASE}/openembedded-core/meta \
  ${MIONBASE}/meta-openembedded/meta-oe \
  ${MIONBASE}/meta-openembedded/meta-networking \
  ${MIONBASE}/meta-openembedded/meta-python \
  ${MIONBASE}/meta-openembedded/meta-filesystems \
  ${MIONBASE}/meta-virtualization \
  ${MIONBASE}/meta-security/meta-tpm \
  ${MIONBASE}/meta-yocto/meta-yocto-bsp \
  ${MIONBASE}/meta-sca \
  ${MIONBASE}/meta-python2 \
  ${MIONBASE}/meta-intel \
  ${MIONBASE}/meta-mion \
  ${MIONBASE}/meta-mion-bsp/meta-mion-${VENDOR} \
  ${MIONBASE}/meta-mion-backports \
"


Note: Just like bash, multi-line strings require having the new line
escaped.

To add the extra layer, all we need to do is add the layer path to BBLAYERS.
Now we can use this by....
Adding Recipes/Packages to an Image

There are a couple of ways of adding recipes to an image. An image has a recipe
for itself and can be added there, or it can be added in mions's local
configuration file (mion/build/conf/local.conf). The difference
between these two being that local.conf can be used for transient changes in
the image's packages or contents, while adding it to an image recipe will mean that
the generated image will always have those included.

You specify a machine name when you build an image.

It would also depend on how significant your changes are. If for example, you just
want to add a piece of already available software, and you don't need it to
be available to anyone else, then local.conf would be a good fit, so long as
you understood that you would have to maintain this file.
However, if your build is more advanced, and you want others to be able to build
it, you should add your own image recipe, along with custom layers (or modifications
to those already existing) as is necessary.

Adding recipes to a mion build adds packages to an image.

If you are an appliance developer, it is most likely that you will have your
own image configuration and so is best added there.

When a package is added to a mion image, the files to be installed are
added directly to the root file system of the operating system. However,
it is possible for the packages to be created as optional installables,
and added to a package feed which enables installed system to optionally
install them at a later date.

While technically, you can maintain your own repository of mion purely
for the purposes of using the local.conf file to modify an image,
consider whether or not it would be more appropriate to have a specific image
recipe.
Permanent Additions

The base image used for APS Networks devices are ONIE compatible and include
ONLPv1. The recipe for this is as follows:
# SPDX-License-Identifier: MIT
require mion-onie-image.inc
require recipes-core/images/mion-image-core.inc

ONLPV="onlpv1"
IMAGE_INSTALL_append = " \
    ${ONLPV} \
    packagegroup-onl-python2 \
"

It includes mion-onie-image.inc which instructs bitbake on how to create an
ONIE image, as well as mion-image-core.inc which sets up the build system for
creating an image, and declares the default packages to be installed.
Otherwise, the recipe is straightforward, after including the aforementioned
files, it merely adds the ONLP and Python2 packages to the image. All you
would need to do is add the name of the package.
For arguments sake, we know that a gcc package already exists; adding it to
the image would be as simple as:
IMAGE_INSTALL_append = " \
    ${ONLPV} \
    packagegroup-onl-python2 \
    gcc \
"


mion/Yocto makes no assumptions about the architecture of the system that
builds are performed on. Further, switch architectures may not be x86_64,
there are, in fact, several arm based units. As no assumptions are made,
when anything is compiled, it is done so as if it were a cross compile,
irrespective of whether the build and target architectures are the same.

Transient Additions

For transient changes, you can modify local.conf which follows a similar
pattern. All we do here is, at the bottom of the file, add an extra
IMAGE_INSTALL option:
IMAGE_INSTALL += " gcc"

Creating a Recipe

A recipe is a file which describes how a piece of software is built and
installed. Recipes can "inherit" from a "bbclass" (BitBake class), which
provides this information in a templated way such that the user written recipe
is much more simple. There are many different classes, but some common ones
are those for projects that use autotools or CMake.
A minimal recipe that builds from a Git repository, may need only contain a
bb file. However, they can also contain patches, as well as complete
source trees. They can also just contain files to be copied directly into
the package. And finally, they could include all of the above.
Adding Sources

A source is a file or repository that is to be used in the build and are
specified in the SRC_URI variable and can either be files or repositories.
Repositories (such as Git, SVN, etc) are pulled using "fetchers". The BitBake
user manual has a list of fetchers and their options
Patches

Examples

The following recipes are examples taken from mion of the various flavours
of recipes you might want to write.

A simple Git recipe: [link, description]
A simple file recipe: [link, description]
Git recipe with patches: [link, description]

As a more detailed example, if we wanted to install some software that
required CMake to be built, we could write a fairly complicated recipe
to perform the build. However there is a bbclass for such projects that
simplifies the desired recipe significantly.
For a searchable list of existing recipes and classes, please visit the
OpenEmbedded layer index.
There is a great deal of documentation on the internet with respect to this
subject, however an example for our PTP software follows:
SUMMARY = "PTP Daemon for last-gen APS Networks Switches"
LICENSE = "CLOSED"

# This is our source code directory. Because the source repository is git, it is
# downloaded to a directory named git, so the location of the source available
# tp BitBake needs to be updated.
S = "${WORKDIR}/git"

SRC_URI = "git://git@github.com/APS-Networks/ptp-daemon.git;protocol=ssh;branch=main"

# We could put in a Git hash, but AUTOREV tells us to grab whatever is at HEAD
SRCREV = "${AUTOREV}"

# We only want to build and install this for the two switch platforms listed
# below; it will not work and when the OS image is built, an error will be
# produced if the target system is not what's expected.
COMPATIBLE_MACHINE="(stordis-bf2556x-1t|stordis-bf6064x-t)"

# This indicates a build time dependency. We need boost to compile some parts
# of the program.
DEPENDS = "boost"

# This is a runtime dependency. It is coupled to the recipe's version number.
RDEPENDS_${PN} = "boost-program-options"

# This is a CMake project, and there exists a bbclass for it. Leveraging this
# means that, assuming the CMake project is set up correctly, there is very
# little more to write.
inherit cmake

# There are some caveats to building with `bitbake`. Not everything has a
# bbclass, one of which being `mkdocs` which our documentation is based on. We
# could write our own recipe, however it is often easier to turn building it
# off. If documentation on the device is needed, then it could be built
# elsewhere, and included in the image as additional filed.
#
# Secondly, BitBake does not play well with projects which themselves in turn
# download other sources. For instance, this `ptp-daemon` software requires the
# `yaml-cpp` and `googletest` provided on GitHub through the `FetchContent`
# CMake module. In ordinary circumstances, this would be downloaded by CMake
# to a subdirectory of the build directory, which is itself a sub-directory of 
# the original checked out sources. However, BitBake will build the project 
# in a folder outside the source directory, which does not play nicely.
#
# You can get this working by setting `${B}` to something like `${S}/git/build`
# however this isn't recommended. The way to do it "idiomatically" is make sure
# that the upstream project can be built without having to download external
# sources. However this is not always possible for every project.
EXTRA_OECMAKE = " -DBUILD_DOCS:bool=false -DBUILD_YOCTO:bool=true"

# This specifies which files are to be installed. These are variables provided
# by Yocto; they are not user specified.
FILES_${PN} = " \
    ${systemd_system_unitdir} \
    ${bindir} \
    ${sysconfdir} \
"

${PN} is a variable available to the recipe containing the recipe version.
This is determined in the name of the recipe. If our bbfile was named
ptp-daemon_0.1.bb, ${PN} would be "0.1".
${S} is the source directory. For GitHub projects, the default will not work as
files are downloaded to a git directory, hence why it is modified.
${B} is the build directory. BitBake prefers that the build directory be
separate from the source.

Building a Recipe on it's Own

It is possible to build a recipe in isolation -- however any of it's
dependencies will have to built as well. This is achieved by calling cronie.sh
with the machine type (which is an absolute necessity), and then providing the
package name:
# From the build directory:
../cronie.sh -m stordis-bf6064x-t ${package}
Debugging

It is often useful to investigate what happened during building the package,
for example, an error occurred, or you want to double check that the produced
package includes all of the expected files.
For files included with the package:
tmp-glibc/work/corei7-64-mion-linux/${package}/${version}-${revision}/image/
Bitbake works by compiling recipes into bash scripts (plus a bit of python) for
each stage in the package generation process.
Both the log files and the scripts can be found in:
tmp-glibc/work/corei7-64-mion-linux/ptp-daemon/${package}/${version}-${revision}/temp/
Modifying Existing Recipes

Existing recipes often modified, usually to add or alter something
that is machine specific. This is performed by way of a BitBake
append file. For example, you might want to remove a compiler flag
(e.g. TARGET_CFLAGS_remove -O2), or add extra files.
If a given package is installed, and there is a bbappend file
with a name
${package}_${version}.bbappend

it will automatically be used when that package version is built. The
append file can also be applied to any version of the package by putting
a wildcard symbol in the place of the version:
${package}_%.bbappend

In this file you can do anything that you would ordinarily do in a
complete recipe. You can also restrict the append file to a
particular platform using
COMPATIBLE_MACHINE = "(stordis-bf2556x-1t|stordis-bf6064x-t)"