Ready, Set, Yocto! (v0.3) : A short guide to getting started with Yocto

ready_set_yocto.txt

================================================================================
                           Ready, Set, Yocto! (v0.3)
                  A short guide to getting started with Yocto 
                      using YP Rocko 2.4 the Raspberry Pi
                                      
                                     jynik 
================================================================================

-------------------------------------------------
 1. Intro
-------------------------------------------------

If you're looking to build Linux-based firmware for embedded platforms, Yocto 
is a good option for you. 

If you're looking to create reproducible and version-controlled automated 
firmware builds with excellent traceability, change tracking, license auditing, 
and configurable QA tests... Yocto is a great option for you.

The purpose of this guide is to get you up and running with a minimal build for
a Raspberry Pi (Zero)  so that you can dive into the documentation while you wait. 

Keep the following at an arms length:

Yocto Quick Start Guide:
http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html

Yocto Mega Manual
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html

Bitbake User Manual
https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html

-------------------------------------------------
 2. Build Environment
-------------------------------------------------

The nature and configuration of your build machine(s) will vary depending your
needs. This guide DOES NOT cover a configuration suitable for production build
deployments. Instead, this covers a "personal dev setup" installed onto a 
portable SSD that can be moved from machine to machine.

My minimum recommended setup is:
    + (X|K)Ubuntu 16.04 or later (I use XUbuntu 17.10 and it works great.)
    + 4 GiB of RAM
    + Intel i5
    + Over 200 GiB of disk space, preferably on an SSD

To enable me to move from machine to machine (e.g., desktop to travel laptop) 
with my build environment, I use a portable SSD with the following restrictions,

    + The SSD is formatted with EXT4
    + All machines mount the SSD to the same exact mountpoint, i.e. "/portable"
    + The User and Group IDs I use between machines match. If you can't 
      guarantee this with already existing users, consider making a new uid and
      gid for your builds.

If you deviate from those restrictions, you'll probably run into problems.


-------------------------------------------------
 3. Dependencies
-------------------------------------------------

You'll need some basics!

~~~
 $ sudo apt install build-essential chrpath gawk git texinfo
~~~


-------------------------------------------------
 4. Build System and Required Layers
-------------------------------------------------

So, let's assume the portable SSD I'm working with is always mounted to a
/portable mountpoint.

Often times I'll be working with multiple version of Yocto, so let's create 
a directory for the current version:

~~~
 $ mkdir -p /portable/yocto/rocko
 $ cd /portable/yocto/rocko
~~~

Let's also make some directories we'll use later:

~~~
 $ mkdir /portable/yocto/rocko/builds
 $ mkdir /portable/yocto/rocko/downloads
~~~

Now lets check out the Rocko branch of "Poky" - Yocto's Build system:

~~~
 $ git clone -b rocko git://git.yoctoproject.org/poky.git
~~~

The bitbake tool used by Yocto parses "recipes" that describe how to fetch,
patch, configure, compile, install, and package software. Collections of related
"recipes" are grouped into "layers" and their names are typically prefixed 
with "meta-".

There's a layer dedicated to Raspberry Pi support. Let's fetch that. Note that
we're still in /portable/yocto/rocko still!

~~~
 $ git clone -b rocko https://github.com/agherzan/meta-raspberrypi.git
~~~

If you take a look at meta-raspberrypi/README.md, you'll note that it has
dependencies on meta-oe, meta-multimedia, meta-networking, and meta-python.
Fortunately, these are all in one repo:

~~~
 $ git clone -b rocko git://git.openembedded.org/meta-openembedded
~~~


-------------------------------------------------
 5. Enter your build environment
-------------------------------------------------

Any time you want to work with the Yocto build system, you'll need your
environment to be in the proper state. A provided script sets everything up for
you.

The argument to this script is the build directory you want to work out of. This
will be created and initialized if it doesn't exist. Lets start a new build
directory called "rpi0":

~~~
 $ source /portable/yocto/rocko/poki/oe-init-build-env /portable/yocto/rocko/builds/rpi0
~~~

You should see some text welcoming you into the environment. You will be placed
into the rpi0 directory that was just created.

There are two files to configure in this directory:

 * conf/local.conf      Build variables and options
 * conf/bblayers.conf   Points Yocto to the meta-* items you downloaded before
 
-------------------------------------------------
 5. Edit conf/local.conf
-------------------------------------------------

First, edit conf/local.conf and make the folowing changes:
 * MACHINE ??= "qemux86" --[to]--> MACHINE = "raspberrypi0"
 * Change #DL_DIR ?= "${TOPDIR}/downloads" --[to]--> DL_DIR = "/portable/yocto/rocko/downloads"

The first item specifies what platform we're targeting. (The name "raspberrypi0"
refers to the "machine" defined by the meta-raspberrypi/conf/machine/raspberrypi0.conf
file.

The second item specifies where we want to store downloaded files. For my own
sanity, I generally have separate per-platform build directories. However, to
save some time, I share the downloads between those builds so that I don't have
to re-download items I already have. This is purely preference.


-------------------------------------------------
 6. Edit conf/bblayers.conf
-------------------------------------------------

Now edit conf/bblayers.conf. We use this file to tell Yocto (well, bitbake)
where to find layers.

Use full paths here and add the paths to  meta-raspberrypi, meta-oe,
meta-multimedia, meta-networking, meta-python. Below is a sample of what your
file will look like:

~~~
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /portable/software/yocto/rocko/poky/meta \
  /portable/software/yocto/rocko/poky/meta-poky \
  /portable/software/yocto/rocko/poky/meta-yocto-bsp \
  /portable/software/yocto/rocko/meta-raspberrypi \
  /portable/software/yocto/rocko/meta-openembedded/meta-oe \
  /portable/software/yocto/rocko/meta-openembedded/meta-python \
  /portable/software/yocto/rocko/meta-openembedded/meta-networking \
  /portable/software/yocto/rocko/meta-openembedded/meta-multimedia \
  /portable/software/yocto/rocko/meta-openembedded/meta-filesystems \
  "
~~~

-------------------------------------------------
 7. Build core-image-minimal
-------------------------------------------------

First, let's build a minimalist Linux setup called "core-image-minimal". This
will boot and drop you into a busybox environment as root. It won't include
much beyond that.

~~~
 $ bitbake core-image-minimal
~~~

And now you wait. The first build will take a lot of time, as you'll be fetching
and building your cross-compiler toolchain, libc, linux, u-boot, an so on...

If you immediately run into an issue, you may have a missing dependency that I
failed to note. Check the log reference in the failure message.

-------------------------------------------------
 8. Copy core-image-minimal to an SD card
-------------------------------------------------

When the build completes, an image that's ready to be dd'd to an SD card will
be in ./tmp/deploy/images

Insert an SD card and write the SD card image to it:

~~~
 $ sudo dd if=./tmp/deploy/images/core-image-minimal-raspberrypi0.rpi-sdimg \ 
		    of=/dev/sdX bs=4M && sync
~~~

-------------------------------------------------
 9. Enable the RPi UART
-------------------------------------------------

Now before you go booting the SD card, we need to make one quick change
that we'll automate later. 

Remount or reinsert the SD card and note the presence of a config.txt on a FAT32
volume. This is some raspberry pi (or broadcom bootloader) shenanigans -- we 
need to enable the console UART and disable some conflicting bluetooth
functionality.

Add the following lines to config.txt:

enable_uart=1
dtoverlay=pi3-disable-bt


Save this file and unmount the SD card.


-------------------------------------------------
 10. Boot core-image-minimal
-------------------------------------------------

Attach a USB UART adapter to the UART pins on the raspberry pi. Here's
a terrible ASCII art diagram. Maybe just Google the pinout instead.

 .-----------------------------      --------------.
 | +5  +5  GND TX  RX                              |
 | 2                                             40|
 |  o   o   x   x   x   o         . . .      o  o  |
 |  o   o   o   o   o   o                    o  o  |
 | 1                                             39|
 |                                                 |
                                  . . .
 |                                                 |
 `--|    |----------------------     --|  |--|  |--` 
     HDMI   		               USB   USB

Plug in in the SD card and power on. You should see boot text and then a login 
prompt akin to the following:

~~~
 Poky (Yocto Project Reference Distro) 2.4.1 raspberrypi0 /dev/ttyAMA0

 raspberrypi0 login: 
~~~

Enter "root" and you'll be logged in. Look in your local.conf for an
"EXTRA_IMAGE_FEATURES" to understand why password-less root login is enabled.


-------------------------------------------------
 11. Boot rpi-hwup-image
-------------------------------------------------

You may notice that lots of Raspberry Pi-specific drivers are missing. That's
because core-image-minimal did not include the platform-specific kernel modules.
The following image recipe adds these to the core-image-minimal build:

			 meta-raspberrypi/recipes-core/images/rpi-hwup-image.bb 

To build this, run:

~~~
 $ bitbake rpi-hwup-image
~~~

When that's done, look for the rpi-hwup-image-raspberrypi0.rpi-sdimg in
./tmp/deploy/images/raspberrypi0/

Go through the same procedure for parts 8-10 to boot this.

-------------------------------------------------
 12. Making your own layer
-------------------------------------------------

Having to manually edit the SD card to add those two tweaks to config.txt is
annoying, so let's use this as an opportunity to learn how to make our own
layer and automate this addition to our image.

To do this, we don't need to be in the bitbake environment that we were
doing builds in. Feel free to open a new terminal.

Let's make a new directory. Let's name our layer "meta-mypi":

~~~
 $ cd /portable/yocto/rocko
 $ /portable/yocto/rocko/poky/scripts/yocto-layer create meta-mypi
~~~

Accept the default value for the "layer priority" when prompted.
Select "n" for the example recipes -- let's not do too many things at once
here. Go back and create a separate layer to work through those later. :)

When done, you should now have a  /portable/yocto/rocko/meta-mypi directory, 
complete with a conf/ directory, an MIT license, and a boilerplate README.

For more info about creating layers, see:
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#understanding-and-creating-layers

-------------------------------------------------
 13. Creating a rpi-config_git.bbappend
-------------------------------------------------

Recipes are stored in files with a .bb extension.

If we want to add "stuff" or otherwise override definitions
in an existing recipe, we can do so by creating a corresponding .bbappend file.

A .bbappend file should have the same name as its .bb counterpart, and should
be located in the same relative path within the layer.

The recipe responsible for creating the config.txt file in the Raspberry Pi
image is: meta-raspberrypi/recipes-bsp/bootfiles/rpi-config_git.bb 

In particular, the do_deploy() function queries variables, which you can set in
your local.conf file, and uses them to toggle various features in a config.txt
pulled from github.com/Evilpaul/RPi-config.git.

What we'll do is add a function that gets run after the do_deploy() and appends
out items to the end of the file.

First, let's create the proper directory structure:

~~~
 $ /portable/yocto/rocko/meta-mypi
 $ mkdir -p recipes-bsp/bootfiles
 $ cd recipes-bsp/bootfiles
~~~

Next, create a rpi-config_git.bbappend file and place the following in it:

~~~
do_deploy_append() {
    echo "" >> ${DEPLOYDIR}/bcm2835-bootfiles/config.txt
    echo "# Enable UART and disable conflicting bluetooth functionality" >> ${DEPLOYDIR}/bcm2835-bootfiles/config.txt
    echo "enable_uart=1" >> ${DEPLOYDIR}/bcm2835-bootfiles/config.txt
    echo "dtoverlay=pi3-disable-bt" >> ${DEPLOYDIR}/bcm2835-bootfiles/config.txt
}
~~~

That's it, as far as the bbappend goes. The last thing to do is enable your new layer in your bblayers.conf.

For more info about recipes and bbappends, check out the trusty MegaManual:

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#new-recipe-writing-a-new-recipe
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#using-bbappend-files


-------------------------------------------------
 14. Build with new layer enabled
-------------------------------------------------

To make use of our bbappend, we have to make our build aware of it.

Let's drop back into our build environment:

~~~
 $ source /portable/yocto/rocko/poki/oe-init-build-env /portable/yocto/rocko/builds/rpi0
~~~


Edit conf/bblayers.conf and append the following:

~~~
/portable/yocto/rocko/meta-mypi
~~~


Now we can build and our changes will take effect:

~~~
bitbake hwup-rpi-image
~~~

When this completes, the SD card image should be updated to contain the new
config.txt file. All successive builds will include your changes to this file.

Note that you can check on the state of the config.txt before you spend time
writing the SD card image. During the build process, this file is written to:

		 ./tmp/deploy/images/raspberrypi0/bcm2835-bootfiles/config.txt

-------------------------------------------------
 15. Final words
-------------------------------------------------

Well, that covers about < 1% of Yocto, but it's a great start on some commodity
hardware.  Having done this, you'll hopefully now have a better sense of what's
discussed in the documentation.

If you ever get yourself into trouble, or find that a particular recipe is in a bad state,
check out the `bitbake cleanstate <recipe>` and `bitbake cleanall <recipe>` commands.

I'll leave it to you to look those up in the reference manual.

Happy hacking!
- jynik