Compiling wireshark 2.6 from source, for and on a Raspberry Pi (September 2018)

WiresharkOnRaspberryPi.md

Compiling wireshark 2.6.4 from source, for and on a Raspberry Pi

Introduction

This is a novice's guide (aka me) to compiling wireshark from source, for and on a Raspberry Pi. Written in tutorial style (ie long and verbose) by me for (a future) me.

I have been using wireshark, and its command line siblings tshark and dumpcap on my Raspberry Pi (model 3B, running Raspbian Stretch) for a few weeks.

You can easily obtain a pre-compiled version using apt-get, and find details about the version using apt show eg

sudo apt-get install wireshark
apt show wireshark

However, as at the time of writing (mid Sept 2018) the version of wireshark that is available from the package repositories used by apt on a Raspberry Pi is 2.2.6

I needed a more up to date version (to get access to an enhancement / bug fix). Although I could find precompiled binaries for other processors, I was unable to find binaries for the armV7 processor used by the Raspberry Pi, so I had to compile a more up to date version (2.6.4) from source. (Had I been brave enough I could probably have gone all the way to the latest "in development" version, in the 2.9 series)

In the end it wasn't as hard as I feared, though there were some things I needed to learn about along the way.

Step 0: Background reading and references

I began with some searching and reading. The wireshark documentation itself is quite helpful, but a couple of other independent blog posts helped me get some preliminaries out of the way.

2013 : Building wireshark from source on debian

• While dated now, it contains useful hints about easy ways of installing all the development tools, libraries and related source code header files that are needed to compile wireshark from source

2017 : Building wireshark for CentOS 7

• In the end I didn't follow the steps from this blog (the instructions in the wireshark manual are more current!) but reading it did help me get an overview of the broad process I would be following.

There are official wireshark documentation instructions about how to build and install wireshark from source in several different places, which was (just slightly) confusing to begin with. My recommendation would be to go to the INSTALL files which are included with the downloaded source code (and of course, on the github repository as well).

2018 : Wireshark wiki build and install directions

• This is now slightly out of date I suspect (eg it references a configure option, --with-ssl which no longer seems to exist.)

2018 : Wireshark users guide - unix build

• Nice concise instructions (that assume everything just works, or you already know enough about compiling and linking programs to figure out what to do if it doesn't)

2018 : Wireshark developers guide

• Equally concise, with some hints about how to change compile time options if you need to.

2018 : Wireshark users guide, troubleshooting

• Some hints about common problems, and what log files to look into if something does go wrong.

2018 : Wireshark2.6 INSTALL file on Github

• I found this the most useful of the references, since it contains more detail about what other libraries and tools you need to compile wireshark from source, how to check you have modern enough versions, and where to update them from if necessary. It also explains some of the available compile time options. NB This (and related) files are included in the downloaded source tarball (or the cloned github repository).

2018 : wireshark-dev mailing list

• When I did strike a problem that stumped me, the people on the wireshark-dev mailing list were most helpful!

There are two options for obtaining the wireshark source code, either

2018 : Wireshark download page

• download a zipped tarball from this wireshark download page, or

2018 : Wireshark github page

• git clone a copy of the source project from this github mirror

Step 1 : Set up the build environment and get dependencies

Compiling wireshark requires :

• a compiler, linker and some other tools
• some (precompiled) libraries from other projects that wireshark uses and
• some header files (aka snippets of source code, usually in ".h" files that are "included" in the wireshark code so it can use those libraries

When I started this was perhaps the most daunting step, but it was in the end quite straightforward. (I may have just been lucky, in that the package archive maintainers for the Raspberry Pi have already compiled an earlier version (2.2.6) of wireshark, so apt already knows about the source build dependencies for wireshark)

First, edit /etc/apt/sources.list so that apt knows about source archives (as well as precompiled binary archives). I simply uncommented the third line of the sources.list file, using

sudo nano /etc/apt/sources.list

then saved it. The uncommented file should look like

deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi

Then I ran :

sudo apt-get update
sudo apt-get build-dep wireshark
# Can't recall whether this final install was essential or not, but I did it anyway
sudo apt-get install  build-essential checkinstall libcurl4-openssl-dev

and waited while a bunch of packages were downloaded, and installed.

Later on, as part of trying to determine the cause of a slight hiccup I was experiencing, I checked that I had appropriate versions of all these dependencies (see the INSTALL file for more information) and everything was fine.

gdb --version
ddd --version
make --version
bash --version
perl --version
bison --version
flex --version
git --version
patch --version

Step 2 : Get the source code for wireshark

First, create a directory to experiment in

cd ~
mkdir wireshark-explore

Initially I downloaded the wireshark tarball, and untarred / unzipped it in my ~/wireshark-explore directory, but for various reasons I later changed to cloning directly from the github repository, so that's what I'll show here.

cd ~/wireshark-explore
git clone -b master-2.6 --single-branch https://github.com/wireshark/wireshark.git

This put the source code into ~/wireshark-explore/wireshark (and assorted sub-directories).

The -b master-2.6 --single-branch meant I only downloaded the source code for the 2.6 version of wireshark (which matches what is available on the download page, as at the time of writing). Omitting that would have downloaded the complete git repository, and by default (unless I did an explicit git checkout) I would have been compiling the latest master version (2.9 series) of wireshark .

Step 3 : Compile wireshark

This was (relatively) straightforward, albeit quite time consuming on the Raspberry Pi. Think in terms of hours of elapsed time, not just "make a cup of coffee". Basically I used the recipe recommended by the wireshark developers, namely use cmake and and ninja.

But before I could do that, I had to install ninja. It turns out that there are two quite distinct packages with ninja in the name available from the Raspberry Pi apt repositories, viz ninja and ninja-build. Anyway, ninja-build is the one I needed (though, once installed, it is referred to simply as ninja).

whereis ninja
whereis ninja-build
apt show ninja-build
apt show ninja
sudo apt-get install ninja-build
whereis ninja

With that out of the way, I could simply set the recipe going, and settle down and wait, and watch the compilation happen, and wait, and wait and ...

# Create a parallel build directory
cd ~/wireshark-explore
mkdir wireshark-ninja
# cd to the build directory
cd ~/wireshark-explore/wireshark-ninja
# Run cmake, specifying ninja as the build system,
# and ../wireshark as the location of the source code
cmake -G Ninja ../wireshark
# Build (compile) the wireshark source code using ninja
ninja

Well almost! There was one final hurdle ahead.

The cmake step was (relatively) quick, and sent a fair bit of output to the console screen.

Once cmake had finished (it produced a file called build.ninja) the ninja step began. ninja is "fast" - given what it is doing - but it had an awful lot to do, so it actually took many hours to complete.

One reason for ninja's speed is that by default it uses multiple cores. The side effect of this on the Raspberry Pi was that everything else became utterly unresponsive - for hours! It was important to not panic, and just leave everything running.

Even then, after several hours, and having completed roughly 1500 of the 3000+ steps required, ninja stopped, and reported that it had encountered an "internal compiler error" (in gcc), about which which I should file a bug report. I did look into what that entailed, but I wasn't really in a position to submit a useful bug report (I certainly hadn't got a minimal reproducible example of the code that generated the error!), so I just started ninja again.

# cd to the build directory
cd ~/wireshark-explore/wireshark-ninja
ninja

ninja recommenced compilation (more or less) where it had stopped, and this time completed roughly 300 of the remaining 1500+ steps before again reporting the "internal compiler error".

So I repeated this procedure several times, eventually getting to the stage where there were only about 300 steps left. However, by this stage, ninja was successfully completing only around 10 steps per attempt before failing, and taking about half an hour per attempt before reporting the error (and before the Raspberry Pi became responsive again).

Thanks to some help from the wireshark-dev mailing list, I learnt that it is possible to get ninja to use only a single core. So I ran

ninja -j1

and this finished all the remaining 300 steps successfully. It still took some time to finish, but no further "internal compiler errors" were encountered.

I speculate (but without firm evidence) that a Raspberry Pi simply just doesn't have sufficient CPU power and other resources to support running multiple large and complex compile and link steps in parallel. I "got away" with parallel compiles for many of the smaller early parts of the wireshark build, and thereby probably saved a lot of elapsed time. But as the process continued and the larger more complex parts of wireshark were being built, the Raspberry Pi needed to go more slowly, and dedicate all the resources it had to a single build step at a time.

Anyway, that's about it. The compiled binaries can be found in the run sub-directory of the wireshark build directory, ie in ~/wireshark-explore/wireshark-ninja/run. I was able to test the newly built version 2.6.4 of wireshark by running

cd ~/wireshark-explore/wireshark-ninja
run/wireshark

Addendum

(A day or so later) I discovered that to be able to run wireshark, tshark etc as the non-root user (by being a member of the wireshark group) - strongly advised for security reasons - I should have used the -DDUMPCAP_INSTALL_OPTION=capabilities option when I ran cmake. See Wireshark Packaging 3.Privileges for a bit more of an explanation.

Luckily this was quite easily and quickly rectified by rerunning cmake and ninja.

cd ~/wireshark-explore/wireshark-ninja
cmake -G Ninja -DDUMPCAP_INSTALL_OPTION=capabilities ../wireshark
ninja

This took only a minute or so to run :-)

Step 4 : Backup the SD card!

The Raspberry Pi SD Card had a lot of new files written to it while wireshark was being compiled. SD Cards can "wear out", and become corrupted, so the next step was to make a backup image of the card. I used Win32DiskImager on my laptop, but you can use whatever method you are familiar with.

Step 5 : Side Steps - aka Troubleshooting notes

All this is written as if it worked (mostly) first time. Of course this isn't true :-( Along the way there were a few problems (principally due to my ignorance and inexperience) that had to be overcome. For example, I originally began by trying to compile wireshark using source code downloaded from the download page, then untarred and unzipped. But the cmake step failed and it took me a while to figure out what was wrong, and how to get round the problem. In fact, I'd never have managed it without assistance from people on the wireshark-dev mailing list. (Aside : the problem had already been fixed in the 2.9 development series of wireshark, and the fix has since been backported to the 2.6 download version).

In hindsight, some quite minor hints would have been quite helpful while I was trying to find out what had gone wrong. They are :

• Obtain a very broad and simplistic understanding of what cmake and ninja are, and how they work; and
• If necessary, ask for help from the wireshark-dev mailing list (the people there were very helpful!)

Cmake and ninja, things I wish I had known more about

Caveat This is my own novice level explanation. It is only intended as a broad and simplistic overview, so it's quite possibly subtly wrong in some aspects - but it should be close enough to show you where to look for further information.

Compiling a complex project like wireshark from source is quite a mammoth task. There are literally thousands of small pieces of code that are compiled, and then linked together into libraries, and finally into executable programs. Most of the work is done by the compiler / linker program gcc. But of course it needs to be told what to do, with what compiler options, and in which order.

Quite a while ago, I had a brief encounter with the standard "magic" incantation for compiling source code under Linux / Unix, viz

./configure
make
make install

In very broad terms

• ./configure probes your system to set everything up, passing information on to the make step via environment variables, and other files it creates, in particular a Makefile. By convention(?), configure reads files which have names ending in .in.
• make reads the Makefile(s) and drives the actual compiling / building process(es)
• finally make install puts the compiled executable code into the standard locations on your computer.

Typically the files ending in .in are themselves generated from various files with endings like .ac, and .am by a suite of programs known collectively as the GNU autotools. And these .ac etc. files are written and maintained by the project developers.

In equally broad terms, cmake and ninja perform the the same functions.

• cmake probes your system to see :

  • what compiler you have, and what options it recognises; and
  • what libraries you have and where the corresponding .h header include files are located;

  then reads a file called CMakeLists.txt, and generates the instruction files for whatever "build system" you have chosen.

  CMakeLists.txt is written and maintained by the project developers, and hence lives in the source directory(ies) - in this example is located at ~/wireshark-explore/wireshark/CMakeLists.txt.

  cmake supports several different "build systems". The default is the "GNU autotools" based system mentioned above, but alternatives can be specified using the -G option, as in this case, where Ninja has been chosen. ninja's design objective is speed!

  When cmake -G Ninja is chosen, cmake produces a build.ninja file in the build directory, - in this example it is located in ~/wireshark-explore/wireshark-ninja/build.ninja

  cmake also produces numerous other files in the build directory, including some log files :

  • ~/wireshark-explore/wireshark-ninja/CMakeFiles/CMakeOutput.log and
  • ~/wireshark-explore/wireshark-ninja/CMakeFiles/CMakeError.log.

  Hint In order to probe the system, cmake runs lots of small test compiles. It is normal and expected for some of these to fail - that is how cmake discovers that a compile option or an optional library is not available. So if you strike a problem during the cmake step - look at the last error message on the console, and the last entries in the log files for hints about what has gone wrong! All the earlier "error" messages are probably just cmake reporting on its normal operations.

• ninja has a "simple" job after all cmake's hard work. It reads build.ninja and compiles and links the executables. In this example, there are over 33,000 lines in build.ninja, so "simple" is a relative term!

  Hint Like most build systems, ninja keeps track of what has already been compiled, so if it falls over or is interrupted for any reason, it is safe to simply restart ninja.

  Hint ninja is fast because it uses multiple cores. If you are building on a low spec machine (like a Raspberry Pi) everything else might become utterly unresponsive while ninja is running, because it is using all the available CPU resources. Don't panic, just wait

  • and then wait some more :-)

  Hint Low spec machines may not be able to complete some of the larger more complex steps while all the cores are in use. In which case try ninja -j1 to use only a single core. (This is speculation on my part, but it worked for me on the Raspberry Pi)

Asking for help from the wireshark-dev mailing list

People on the mailing list were most helpful indeed - THANKS

The usual rules apply

• check existing answers before asking a new question;
• be as specific as you can be about your problem;
• provide a sensible amount of background information about what you've done; and of course,
• be polite.

A couple of minor mistakes which I could have avoided were :

• I should have signed up to the wireshark-dev mailing list before sending my first email to wireshark-dev@wireshark.org - it would have saved a moderator the effort of reviewing my email! ; and
• I should not have ticked the email digest option when I did sign up (it made responding to the suggestions I received just a tiny bit harder than it should have been).